Can Large Multimodal Models Uncover Deep Semantics Behind Images? (2024)

Abstract: Understanding the deep semantics of images is essential in the era dominated by social media. However, current research works primarily on the superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantics. In this work, we introduce DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models’ (LMMs) capacities of visual deep semantics. DEEPEVAL includes human-annotated dataset and three progressive subtasks: fine-grained description selection, in-depth title matching, and deep semantics understanding. Utilizing DEEPEVAL, we evaluate 9 open-source LMMs and GPT-4V(ision). Our evaluation demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans. For example, GPT-4V is 30% behind humans in understanding deep semantics, even though it achieves human-comparable performance in image description. Further analysis reveals that LMM performance on DEEPEVAL varies according to the specific facets of deep semantics explored, indicating the fundamental challenges remaining in developing LMMs.

## introduction the image is more than an idea. it is a vortex or cluster of fused ideas and is endowed with energy. however, previous efforts in visual understanding mainly focus on surface-level aspects of images, such as object attributes @xcite and relationship reasoning @xcite . earlier attempts on deep semantic are limited in scope, focusing solely on sarcasm or humor, @xcite @xcite , and lack in systematic investigation of the inherent deep semantic. to address the mentioned limitations and fill the current research gap, we introduce deepeval, a benchmark for understanding the deep semantics of cartoons across various categories, accompanied by a meticulously annotated dataset. additionally, we introduce three tasks: fine-grained description selection, in-depth title matching, and deep semantics understanding, to comprehensively evaluate models' capabilities in understanding deep semantics. cartoons, often imbued with profound 1 1898 meanings by their creators, are an ideal subject for this study. the deepeval dataset comprises over 1,000 samples, each featuring a cartoon image and manually annotated components, including image description, title, and deep semantics. moreover, we develop multiple-choice questions for quantitative assessment, tailored for each task. we conduct evaluations on various open-source lmms as well as the proprietary gpt-4v(ision). our findings reveal a significant gap between the capabilities of ai and humans in understanding deep semantics. models with a larger number of parameters generally demonstrate a better understanding of deep semantics. moreover, we discover that incorporating a description significantly helps these models in grasping the underlying semantics of an image. furthermore, we also explore the performance of different models across various categories of images. by undertaking deepeval, our goal is to promote research in model development, focusing on a deeper understanding of semantics in visual content. visual deep semantics understanding understanding the deep semantics of visual contents represents a critical cognitive ability in humans. for ai, this ability showcases its depth of understanding images @xcite . present evaluations @xcite @xcite @xcite @xcite ) mainly concentrate on superficial aspect of understanding. pioneering works in affective image classification @xcite have shown that lmms are capable of attaining an understanding beyond mere surface content. research in sarcasm @xcite @xcite and humor detection @xcite only employs classification tasks. further work @xcite provides explanations for satirical content. the most relevant prior work @xcite selects humorous captions for images and provides explanations. however, it exclusively focuses on humor evaluation. in contrast, our work is pioneering in its comprehensive exploration of visual deep semantics across multiple categories, offering a more thorough assessment of the deep semantics within images. we provide a detailed comparison between our method and previous studies in ## dataset and task overview the deepeval dataset includes 1,001 samples, each with an image and three manually annotated components: a description, a title, and deep semantics. the statistical information about the text is displayed in the others serve as distractors. figure 1 illustrates examples of the manually annotated components and the multiple-choice questions. to explore the capabilities of lmms in comprehending the deep semantics of image, we construct a comprehensive evaluation consisting of three main subtasks: evaluating the ability of models to accurately identify the surface-level details of images. together, these subtasks offer a robust and multifaceted evaluation of llms, enabling a deeper understanding of their strengths and limitations in image understanding. ## dataset construction we collect deepeval dataset in a multi-step crowd-sourcing pipeline, including 1) image col-lection, 2) data annotation, 3) option generation. with selected high-quality cartoon images, we ask crowd-source workers to write a description, a title and deep semantics of each image. ## 1903 answer: a (a) tell us not to feel that our parents have not given us enough, they have already given everything they have. ## conclusion we propose deepeval, a benchmark for visual deep semantics of lmms. deepeval consists of well-annotated dataset and three subtasks: finegrained description selection, in-deep title matching, and deep semantic understanding. evaluations are conducted on the leading lmms, revealing a significant gap between ai and human capabilities in understanding deep semantics. further analysis indicates that integrating image descriptions during the inference process enhances lmms' ability to perceive deep semantics. the model's ability to understand deep semantics also improves as the 8 number of parameters increases. furthermore, our dataset is divided into multiple categories, and we conduct a more detailed analysis within these categories. existing models still have a long way to go in terms of visual deep semantics understanding compared to humans. we hope that the proposed dataset and tasks can pave the way for ai to achieve a deeper understanding of the profound semantics conveyed by images.