VGA : Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning (2024)

Abstract: Large Vision-Language Models (VLMs) have already been applied to the understanding of Graphical User Interfaces (GUIs) and have achieved notable results. However, existing VLMs often overly rely on internal text-based knowledge while neglecting visual inputs. This imbalance may lead models to produce answers that do not align with the visual content in GUI comprehension tasks. Such inaccuracies are termed as ‘hallucinations’ where models generate incorrect or illogical responses upon visual verification against GUI elements. These errors result in misinterpretations and diminish the model’s practical utility in applied settings. To address these issues, we introduce VGA, a fine-tuned model designed for comprehensive GUI understanding. Our model aims to balance attention image and text to enhance interpretation and reduce hallucinations. We construct a Vision Question Answering (VQA) dataset of 63.8k high-quality examples with our propose *Referent Method*, focusing on response with visual content of images. We then design a two-stage fine-tuning method to enhance both the model’s accuracy to extract information from image content and alignment with human intent. Experiments show that our approach enhances the model’s ability to extract information from images and achieves state-of-the-art results in GUI understanding tasks. https://github.com/Linziyang1999/VGA-visual-GUI-assistant

## introduction large vision-language models (vlms) have recently emerged as a powerful approach for various multimodal tasks. these models acquire textual knowledge through pre-training and develop image understanding abilities during instruction-tuning @xcite . vlms can * corresponding author effectively process both visual and linguistic information by using a visual-language projector, which maps different types of data into a shared latent space. for instance, llava @xcite and instructblip @xcite have demonstrated strong zero-shot capabilities in tasks like image captioning, visual reasoning, and complex conversations. unichart @xcite , on the other hand, has demonstrated the powerful capabilities of vlms in the understanding of formatted charts. with the proliferation of mobile applications, the importance of graphical user interfaces (guis), which serve as a critical bridge between end users and applications, has increasingly garnered scholarly attention. guis, characterized by their structured layouts, rich graphical and textual content, and the inclusion of human operational logic @xcite , present a complex challenge: can the success of vlms be applied to the gui domain? traditional gui comprehension method focus on conveying the actual user interface interactions and mirror the user's direct experience with the gui. for instance, labeldroid @xcite utilizes deep learning to predict labels for imagebased buttons from a variety of commercial apps available on google play. similarly, tango @xcite employs custom computer vision (cv) and text retrieval techniques to analyze visual and textual information on mobile screens. however, traditional visual gui comprehension methods focus mainly on identifying gui components but fail to fully comprehend the graphical and textual information, layout, and the interaction context within the interface. in contrast, modern approaches such as those employed by ferret-ui and cogagent, which utilize lvlms, demonstrate significant advantages in this domain. these methods are designed to process both detailed textual and graphical content, leveraging their pre-trained capabilities. nevertheless, both ferret-ui @xcite to address these issues, we propose vga, a model fine-tuned on a self-construct 63.8k dataset, using a novel training method we design. during the dataset construction, we employ knowledge distillation from large language models to construct 63.8k dataset based on rico @xcite , and we adopt the referent method to enhance the model's focus on image content by using visual and position information in constructing dataset, thereby increasing the relevance and accuracy of the responses. in the training process, we apply a foundation and advanced comprehension (fac) approach: the foundation stage enhances the model's understanding of gui image, while the advanced comprehension stage improves the model's ability to respond to complex questions based on its understanding of the gui. additionally, we employ a task progression and reinforcement approach to create logical chains between tasks and within responses to strengthens the model's ability to reason and infer relationships within the provided context. through these integrated strategies, vga achieves state-of-theart performance in gui comprehension tasks. our primary contributions are as follows: ## problem in gui comprehension the primary cause of hallucinations in vision-language models (vlms) can be attributed to their reliance on response patterns learned from traditional large language models (llms). in this section, we investigate the occurrence of hallucinations in contemporary gui models and propose a method to reduce these inaccuracies. ## gui comprehension dataset to address the imbalance between image and text in gui analysis, we propose a high-quality comprehension dataset tailored for lvlm training. this dataset includes detailed annotations of both textual and visual elements, ensuring a balanced and comprehensive training environment. ## tuning script existing fine-tuning methods in lvlm treat question tokens and image tokens equally, which neglects the need for lvlm to prioritize image information. as a result, models may understand tasks well but generate responses based solely on questions. this results in hallucinations in gui comprehension (see foundation stage: the foundation task dataset is an instructionfollowing dataset that fix the command and response formats. this stage incorporates direct visual information and employs a fixed format for questions and responses to train the model to correlate with image content. the fixed format is intended to "freeze" the variations in questioning and responding styles, focusing the model's learning process on understanding and interpreting visual attributes effectively. this controlled environment ensures that the model develops a robust capability to recognize and interpret visual data independently from textual content. ## conclusion we introduce vga, a fine-tuned model for gui comprehension, leveraging a custom dataset and novel fine-tuning method. the dataset, constructed using a reference method, focuses on imagecentric responses. the training can be divided into two stage: foundational skill alignment and human intent alignment. the foundation stage emphasizes learning the relationship between images and responses, while the advanced stage focuses on intent following. ablation studies confirm the efficacy of our training method. we believe that our training data and method will serve as valuable resources for future research.