Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLM s (2024)

Abstract: Multimodal Large Language Models (MLLMs) demonstrate a strong understanding of the real world and can even handle complex tasks. However, they still fail on some straightforward visual question-answering (VQA) problems. This paper dives deeper into this issue, revealing that models tend to err when answering easy questions (e.g., Yes/No questions) about an image, even though they can correctly describe it.We refer to this model behavior discrepancy between difficult and simple questions as model laziness.To systematically investigate model laziness, we manually construct LazyBench, a benchmark that includes Yes/No, multiple choice, short answer questions, and image description tasks that are related to the same subjects in the images.Based on LazyBench. we observe that laziness widely exists in current advanced MLLMs (e.g., GPT-4o, Gemini-1.5-pro, Claude 3, LLaVA-1.5, LLaVA-1.6, and QWen-VL). We also analyzed the failure cases of LLaVA-1.5-13B on the VQA-v2 benchmark and discovered that about half of these failures are due to the model’s laziness. This further highlights the importance of ensuring that the model fully utilizes its capability.To this end, we conduct a preliminary exploration of how to mitigate laziness and find that chain of thought can effectively avoid this issue. The data can be accessed at https://github.com/Akutagawa1998/LazyBench.

## introduction multimodal large language models (mllms) @xcite integrate multimodal content such as images into large language models (llms) @xcite . represented by openai's @xcite , mllms have demonstrated impressive capabilities across various complex multimodal tasks (openai, 2023a; yang et al., † pinjia he is the corresponding author. 2023). however, existing research indicates that even state-of-the-art mllms still suffer from some straightforward visual questions (e.g., "is the door of the truck cab open?" for an image of a truck.) @xcite . a natural question arises: why do mllms struggle with these easy questions? in this work, we dive deeper to explore this question and find that mllms often struggle with simple questions (like yes/no questions) about an image, even though they can accurately describe the image itself. for example, as present in figure 1 , when we asked gpt-4v, "is this man wearing a beige cap?" gpt-4v answered "yes", which is incorrect. in contrast, when we asked it a similar but more difficult question, "please describe the cap that the man is wearing", gpt-4v correctly described its color. in this paper, we describe this phenomenon where mllms perform well on the description tasks but make mistakes on simpler tasks as model laziness 1 . to systematically study model laziness, we manually construct a benchmark called lazybench. we found image pairs encoded as "similar images" by the pretrained contrastive language-image pre-training (clip) @xcite model and designed simple yes/no questions on their visual differences. we collected images from where gpt-4v (website version) fails in the above-mentioned questions. then for each image, we handcraft three different types of questions about the same subject of the yes/no question: multiple choice, short answer question, and a description task. we use lazybench to evaluate advanced closed-source models like gpt-4o, gpt-4v (openai, 2023a), gemini-1.5-pro @xcite , and claude 3 @xcite , and open-source models like figure 1 : mllms sometimes fail to correctly answer straightforward yes/no or multiple-choice questions based on images. however, they often manage to avoid these errors when describing the images. we refer to this phenomenon as "model laziness." llava-1.5 @xcite , llava-1.6 @xcite and qwen-vl @xcite . the results show that these state-of-the-art mllms significantly suffer from laziness: they show a low accuracy on yes/no questions (e.g., gpt-4v: 28.72%, claude 3: 34.66%), and multiple choice questions (e.g., gpt-4v: 54.45%, claude 3: 55.45%), while performing significantly better on the corresponding description tasks (e.g., gpt-4v: 71.28%, claude 3: 57.43%). we further explore to what extent mllm laziness is prevalent in the widely used visual questionanswering (vqa) benchmarks. to this end, we propose a simple llm-based framework that automatically evaluates the extent of laziness in their failure cases. we find that 41.15% failure cases of llava-1.5-13b on vqa-v2 are caused by model laziness. we believe this provides valuable insights into the way to improve the capability of mllms: in addition to allowing mllms to learn more knowledge, it is equally important to ensure that mllms are fully utilizing the knowledge learned. to mitigate the influence caused by model laziness in simple tasks, we implemented a chain of thought (cot) @xcite based method to make the task "harder". we require mllms to handle the description task first before answering a yes/no or a multiple-choice question. the results show that our method fixed around 40% cases of laziness and effectively improved mllms' performance in those tasks. in summary, our contributions are listed below: ## mllms are being lazy to thoroughly understand and analyze the lazy phenomenon, where mllms perform well on descriptive tasks but fail on simple tasks, we construct the lazybench benchmark. therefore, in this section, we first introduce the methods and steps for constructing lazybench. subsequently, we measure the extent of the lazy phenomenon of current stateof-the-art mllms on lazybench. finally, we use a cot-based method to mitigate the mllms laziness. ## conclusion this paper highlights the laziness in mllms: a model can handle difficult tasks (e.g., describe the subject) but fails on simple tasks (e.g., a corresponding yes/no question). we provide a benchmark lazybench that systematically shows this discrepancy in model performance across advanced mllms. our findings indicate that in addition to allowing the model to learn more knowledge, it is equally important to ensure that mllm is fully utilizing the knowledge learned.