Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models (2024)

Abstract: To enhance a question-answering system for automotive drivers, we tackle the problem of automatic generation of icon image descriptions. The descriptions can match the driver’s query about the icon appearing on the dashboard and tell the driver what is happening so that they may take an appropriate action. We use three state-of-the-art large vision-language models to generate both visual and functional descriptions based on the icon image and its context information in the car manual. Both zero-shot and few-shot prompts are used. We create a dataset containing over 400 icons with their ground-truth descriptions and use it to evaluate model-generated descriptions across several performance metrics. Our evaluation shows that two of these models (GPT-4o and Claude 3.5) performed well on this task, while the third model (LLaVA-NEXT) performs poorly.

## introduction vehicle dashboard icons convey critical information to drivers, who must quickly understand the symbols' meaning and take appropriate action. however, many drivers are unfamiliar with these icons, emphasizing the pressing need for a virtual assistant that can explain the icons' meanings. for example, when presented with a dashboard icon resembling a steaming cup, the driver might naturally ask "what does that cup icon mean?" the correct response is that this is a warning from the vehicle's driver attention system. the inago netpeople® assistant is a proprietary voice-based virtual assistant platform for automotive drivers. it can answer drivers' questions based on knowledge extracted from text documents, such as car manuals. however, netpeople currently struggles with icon-related inquiries because its text-based knowledge base lacks icon descriptions. this gap means driver's questions about icons cannot be matched to any existing knowledge items. currently, no conversational system for drivers can answer questions about dashboard icons. to address this, we aim to automatically generate text descriptions for icon images, enabling netpeople to include questions and answers (qas) about dashboard icons in its knowledge base. this task presents several challenges. first, existing image description systems are trained mainly on natural images, whereas icon images are drawings. second, understanding an icon's function, beyond its visual description, requires context from the manual and is harder than typical image captioning. training and evaluating a model that generates both visual and functional icon descriptions necessitates a labeled dataset, which currently does not exist. third, while many metrics for text generation are available, identifying the most suitable metrics for evaluating both visual and functional descriptions of dashboard icons is crucial. in this work, we compile a dataset of 408 vehicle dashboard icon images and their corresponding names/functions, which we collected from 42 vehicle manuals. we use state-of-the-art multimodal large vision-language models (lvlms) (i.e., @xcite , llava-next @xcite and claude 3.5 @xcite ) to generate natural english descriptions of each icon's visual design and function. such descriptions can form qa pairs for netpeople's knowledge base. we assess model performance using standard performance metrics and human evaluation. the key contributions of this work are: ## related work efforts to verbalize images through image captioning @xcite and summarization (celis and keswani, 2020) use retrieval @xcite or generation methods @xcite . recently, researchers have combined these by retrieving image-caption pairs and inputting them into generation models @xcite . lvlms use large language models (llms) in vision-language tasks either as schedulers @xcite , where the llm manages various visual models as plug-and-play modules based on specific task requirements, or as decoders @xcite , enabling cross-modal knowledge transfer. with the increasing need for larger language model backends, approaches like instructblip @xcite and llava @xcite collect extensive human instruction datasets to train larger lvlms. these models then undergo end-to-end training, enhancing the llms with visual reasoning capabilities. @xcite and claude 3.5 @xcite are advanced multimodal models that process text and image inputs to generate text outputs, exhibiting human-level performance on various professional and academic benchmarks. in this work, we utilize three recent lvlms, i.e., gpt-4o, claude 3.5, and llava-next, to generate descriptions of dashboard indicator icons. our goal is to enable a qa system to answer drivers' questions about these icons. to the best of our knowledge, this is the first application of ai models for generating such descriptions. ## evaluation in our evaluation, we primarily considered the performance of different lvlms, effectiveness of fewshot prompting, and correlation of automated performance metrics with human evaluation. we also evaluated the impact of description type and input type on model performance. ## conclusion we have presented a novel application of large vision-language models to generation of vehicle dashboard icon descriptions. our contributions include a novel task for automatic generation of visual and functional descriptions of automotive icons, enabling qa systems to answer questions about dashboard icons, which existing in-car qa systems currently do not. we created a novel dataset consisting of 408 different icons from four different vehicle manufacturers for this specific domain and provided insights into challenges and performance in an automotive context. the impact includes improved driver safety through reduced cognitive load, as drivers can quickly access clear explanations of unfamiliar icons without manual distraction. our work furthermore assists the development of easier to use and more powerful vehicle assistants, which benefits drivers with varying levels of automotive knowledge. beyond driver assistance, our methodology and findings may have broader applications in evaluating lvlm performance on abstract or symbolic images across various domains, such as industrial design or medical imaging. for future work, we plan to fine-tune llava using our dataset, focusing on improving the vision encoder to better differentiate between icons and reduce hallucinations. we will explore replacing llava's original clip vit-l vision encoder with more capable versions, such as those using data filtering networks (dfn) @xcite and quick gelu @xcite , which have shown improved performance on imagenet. additionally, we aim to develop new metrics that are more responsive to hallucinations in generated image descriptions, such as visual-likeness aware named entity similarity. this approach would capture that semantically different objects (e.g., a tire cross-section and a horseshoe) may describe similar shapes, while semantically close items (e.g., a brake pedal and a brake disc) may be visually distinct. we also plan to expand the dataset by processing additional vehicle manuals and collecting more human-generated descriptions. with this study, we lay the groundwork for improved icon interpretation in conversational driver assistance systems and hope to contribute to the development of more effective and user-friendly automotive interfaces.