I ntelli C ockpit B ench: A Comprehensive Benchmark to Evaluate VLM s for Intelligent Cockpit (2025)

Abstract: The integration of sophisticated Vision-Language Models (VLMs) in vehicular systems is revolutionizing vehicle interaction and safety, performing tasks such as Visual Question Answering (VQA). However, a critical gap persists due to the lack of a comprehensive benchmark for multimodal VQA models in vehicular scenarios. To address this, we propose IntelliCockpitBench, a benchmark that encompasses diverse automotive scenarios. It includes images from front, side, and rear cameras, various road types, weather conditions, and interior views, integrating data from both moving and stationary states. Notably, all images and queries in the benchmark are verified for high levels of authenticity, ensuring the data accurately reflects real-world conditions. A sophisticated scoring methodology combining human and model-generated assessments enhances reliability and consistency. Our contributions include a diverse and authentic dataset for automotive VQA and a robust evaluation metric aligning human and machine assessments. All code and data can be found at https://github.com/Lane315/IntelliCockpitBench .

## introduction in recent years, with the advancement of visual language models (vlms) @xcite @xcite , intelligent cockpit technology has made significant progress, becoming an important interface for the next generation of humancomputer interaction. subsequently, benchmarks like drivebench @xcite and nuscenes-qa @xcite have been proposed to evaluate the visual question-answering (vqa) capabilities in autonomous driving scenarios. even figure 1 : the relationship between model size and score in english queries across various vlms on intellicockpitbench. notable models such as gpt-4o @xcite and gemini-2.0-flash @xcite are distinguished by their superior performance despite larger sizes. the dotted line represents an estimated trend indicating the positive correlation between model size and performance. so, these benchmarks remain primarily focused on decision-making scenarios such as autonomous driving and do not adequately consider nondecision-making scenarios aimed at enhancing user experience and interaction. this has significant limitations in the research field. limitation 1: the lack of comprehensive benchmarks specifically designed to evaluate the performance of vlms in non-decision-making scenarios within intelligent cockpits. limitation 2: existing gpt-based @xcite automatic evaluation methods typically rely on uniform assessment standards, which overlook the specific nature and requirements of different queries. this further emphasizes the necessity of developing evaluation benchmarks tailored to different queries types. to address these limitations, we propose a comprehensive benchmark named intellicockpitbench to evaluate vlms for intelligent cockpits. this benchmark includes a diverse collection of images captured from front, side, and rear cameras, encompassing various road types and weather conditions to provide a comprehensive external perspective. additionally, intellicockpitbench features interior images to reflect the complexity of the in-vehicle environment. the curated dataset also integrates data from both moving and stopping vehicle states, ensuring a thorough representation of real-world scenarios @xcite . taking into account the scenarios of visual information augmented, we have also implemented data augmentation techniques to ensure the robustness of intellicockpitbench in various unexpected situations. all queries in our dataset are collected through driver surveys and generalized using gpt-4o @xcite to ensure their authenticity and diversity. note that all included images and queries are verified for high levels of authenticity and have undergone human review, which ensures that the data accurately reflects real-world driving scenarios. furthermore, we design three key llm-as-ajudge methods including chain-of-thought reasoning, multi-dimensional variance analysis, and rule-calibrated referencing. this evaluation method not only defines different evaluation metrics for various queries but also assigns importance scores to these metrics. additionally, it utilizes chain-of-thought to generate explanations and final ratings, ensuring both high reliability and interpretability. as shown in figure 1 and overall, our key contributions are as follows: ## intellicockpitbench in this section, we introduce an overview of the data composition, the dataset construction, and the evaluation paradigm of intellicockpitbench. ## experiment in this section, we conduct extensive benchmark experiments and analyses in intellicockpitbench, providing detailed discussions that lead to our observations and conclusions step by step. ## conclusion in this paper, we present intellicockpitbench, a comprehensive benchmark designed specifically to evaluate vlms for the intelligent cockpit. this benchmark addresses a significant gap in multimodal vqa research by incorporating a diverse and representative dataset that includes various image perspectives and four driving scenarios. we propose three innovative evaluation methods and use them to evaluate 15 vlms. experimental results demonstrate that gpt-4o performs well but all models struggle with complex reasoning tasks.