MACAROON : Training Vision-Language Models To Be Your Engaged Partners (2024)

Abstract: Large vision-language models (LVLMs), while proficient in following instructions and responding to diverse questions, invariably generate detailed responses even when questions are ambiguous or unanswerable, leading to hallucinations and bias issues. Thus, it is essential for LVLMs to proactively engage with humans to ask for clarifications or additional information for better responses. In this study, we aim to shift LVLMs from passive answer providers to proactive engaged partners. We begin by establishing a three-tiered hierarchy for questions of invalid, ambiguous, and personalizable nature to measure the proactive engagement capabilities of LVLMs. Utilizing this hierarchy, we create PIE, (ProactIve Engagement Evaluation) through GPT-4o and human annotators, consisting of 853 questions across six distinct, fine-grained question types that are verified by human annotators and accompanied with well-defined metrics. Our evaluations on indicate poor performance of existing LVLMs, with the best-performing open-weights model only achieving an Aggregate Align Rate (AAR) of 0.28. In response, we introduce MACAROON, self-iMaginAtion for ContrAstive pReference OptimizatiON, which instructs LVLMs to autonomously generate contrastive response pairs for unlabeled questions given the task description and human-crafted criteria. Then, the self-imagined data is formatted for conditional reinforcement learning. Experimental results show MACAROON effectively improves LVLMs’ capabilities to be proactively engaged (0.84 AAR) while maintaining comparable performance on general tasks.

## introduction large vision-language models (lvlms) demonstrate remarkable capabilities in multimodal tasks * work was done while shujin wu was an intern at the university of illinois urbana-champaign. 1 the code is made public at https://github.com/ shujinwu-0814/macaroon . what color is the hair of the person in the image? the person in the picture has blond hair. there are two persons in the image, may i know which one you are referring to specifically? human will say: llava says: question: is the person in orange pants male or female? the person in orange pants is male. sorry, i'm not able to determine the gender of that person from the image alone. requiring both visual understanding and language processing @xcite @xcite . however, their constant preparedness to deliver information causes them to become passive answer providers at all times: lvlms invariably generate detailed and firm responses, even when the given question is ambiguous or unanswerable. for example, in figure 1 , when faced with unclear or invalid questions, one of the best-performing open-weight lvlms, llava @xcite , tends to make unsupported assumptions, resulting in biased and hallucinated responses. this tendency largely stems from a lack of proactive engagement, which should ideally include challenging invalid questions, requesting clarifications on ambiguous questions, and seeking additional information when necessary. to systematically assess a model's engagement ability, we design a three-tiered structured hierarchy of question types, reflecting three types of desired behavior: 1) tier-i invalid questions assess the ability of lvlms' to identify and dismiss unanswerable questions or those based on false premises, establishing a foundation for reliable ai reasoning. 2) tier-ii ambiguous questions assess lvlms' capacity to request clarifications for en-hancing human-ai interactions and lvlm utility. building on top of this hierarchy, we create the proactive engagement benchmark pie. pie contains 853 image-question pairs, each meticulously verified by human annotators. we construct the dataset by first instructing gpt-4o @xcite with human-written criteria for question generation and best question selection. further, human annotators are asked to examine each image-question pair and only collect high-quality instances as our evaluation dataset. for evaluation, we define aggregated align rate (aar), calculated as the macro average ratio of questions for which the evaluated model's response fully align with the human expectations over three tiers. our initial evaluations using pie reveal that even the most advanced open-weights models suffer from a significant gap between their current capabilities and the nuanced requirements of effective humanmodel interaction (0.28 aar for llava). to bridge this gap, we propose self-imagination for contrastive preference optimization, abbreviated as macaroon, to enhance the proactive conversation capabilities of lvlms. macaroon operates by first directing lvlms to produce contrastive response pairs based on the task description and human-crafted criteria. this data subsequently facilitates conditional reinforcement learning @xcite , enabling lvlms to differentiate between effective and ineffective responses and unifying the training data format. the experimental results of macaroon indicate a promising shift in the behaviors of lvlms, manifesting a more dynamic and proactive engagement paradigm (0.84 aar after macaroon). further, we show that macaroon enables lvlms to generate responses more tailored to humans by proactively eliciting preferences during initial interactions. our contributions are summarized as follows: ## measuring proactive engagement: pie previous benchmarks @xcite @xcite primarily assess the general multi-modal understanding and reasoning capabilities of lvlms by measuring questionanswering accuracy. to formally evaluate how well models perform in terms of proactive engagement, we first break down the engagement capability of lvlms into three tiers: challenging invalid question settings, seeking clarifications, and uncovering latent human preferences through interactive conversations. based on these aspects, we create a new benchmark pie. in this section, we describe our dataset construction process and the metrics design. ## macaroon we introduce macaroon, self-imagination for contrastive preference optimization, designed to enhance the proactive engagement capabilities of lvlms. the illustration of macaroon is depicted in figure 6 . we start by outlining our method for constructing a preference dataset through selfimagination. next, we detail the training algorithm that effectively utilizes this dataset and the inference-time strategy. finally, we describe the implementation details.