B loom VQA : Assessing Hierarchical Multi-modal Comprehension (2024)

Abstract: We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom’s Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and also shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria. The dataset can be accessed at https://huggingface.co/datasets/ygong/BloomVQA.

## introduction recent advances of machine intelligence solutions have demonstrated tremendous success in a wide range of language and multi-modal tasks over diverse domains @xcite @xcite @xcite . with increasing popularity of such solutions, how to systematically evaluate and analyze the models remains a key concern for confident application. many recent efforts have focused on probing the capabilities and risks of the models based on different tasks, measures and perspectives @xcite @xcite . for example, recent studies have explored functionalities and limitations of large language models (llms) from the perspectives of cognitive science @xcite and semantic consistency @xcite . in the context of visual question answering (vqa), several benchmarks target demonstration of multi-modal reasoning and comprehension beyond mere memorization of low-level statistical patterns in the data. for example, gqa @xcite leverages scene graph representations to construct vqa tasks describing reasoning based on relations of different entities in single-frame images. other datasets @xcite leverage sequential visual inputs to form evaluations addressing procedural tasks or story comprehension over longer time horizons. however, little effort has been made in curating datasets to support systematic evalua-figure 2 : graded evaluation on bloomvqa data following bloom's taxonomy @xcite . for vlp models, the vqa accuracy decreases as the task level increases, while the qa accuracy using no visual inputs remains low. for gpt-4v, the vqa accuracy greatly improves over all levels while the comparison to qa accuracy suggests that the model tends to either bypass or even get confused by visual contents especially at high levels. tion of multi-modal comprehension following a theoretically-grounded definition of comprehension, leading to tasks describing only a limited set of cognitive processes at relatively low levels. motivated by this challenge, we draw inspiration from theories and methodologies developed by education researchers on k-12 reading comprehension, where comprehension tasks are graded based on underlying cognitive processes involved following a taxonomy named bloom's taxonomy @xcite . in bloom's taxonomy (figure 2 ), handling of knowledge is categorized into 6 levels from the most basic (level 1) to the most advanced (level 6), including "remember," "understand," "apply," "analyze," "evaluate" and "create." for each level, a group of cognitive skills are identified as required to handle tasks at the corresponding level. for example, the skill of differentiating is required to achieve successful analysis (level 4), while the skill of critiquing is required to achieve successful evaluation (level 5). following this taxonomy, we propose bloomvqa dataset containing vqa tasks categorized based on the level of cognitive skills required. to support a wide range of tasks reflecting basic and advanced comprehension, we collect multiple-choice vqa samples with open-ended questions and free-form answer choices based on picture stories designed for early childhood education. serving as an evaluation set for multimodal comprehension, bloomvqa contains 1200 core data samples collected from human annotators based on 20 stories, categorized into 6 bloom's levels. the core data samples are further augmented based on the hierarchical structure of bloom's taxonomy. specifically, we generalize the concept of scene graphs and propose a novel graph representation named the story graph (figure 1 ). while scene graphs focus on attributes and relations of low-level entities mapped to knowledge mostly at level 1 in bloom's taxonomy, to accommodate advanced tasks at higher bloom's levels, we consider different events, each of which can presumably be described by a separate scene graph, as the nodes of a story graph. we consider edges describing relations between pairs of events. in addition to low-level relations (e.g., temporal relations), we describe relations corresponding to underlying cognitive processes (e.g., making inference, making prediction) which link different events in reaching an overall comprehension of the story. in this way, given a vqa task constructed about one event, new vqa tasks can be automatically constructed at a combinatorial scale by traversing the underlying story graph, as information recorded along the path of traversal can be introduced as the context in forming new vqa tasks. bloomvqa can be used as a benchmark for multi-modal comprehension engines including current multi-modal llms @xcite @xcite , as it enables systematic evaluation of model capabilities with quantitative metrics characterizing its alignment to human comprehension, which is an important aspect for models to approach artificial general intelligence (agi) @xcite . first of all, with data categorized based on bloom's levels, we can probe models in a graded manner and examine model behaviors on tasks posing different levels of challenges for human comprehension, especially including tasks requiring high-level cognitive skills. furthermore, we propose novel metrics characterizing the consistency of model performance following the underlying hierarchy and examine intuitive hypotheses mapping model behaviors to human comprehension patterns. in this work, we examine the hypothesis that models succeed at more challenging tasks are less likely to fail at easy ones. we also study how relevant knowledge, where the relevancy can be defined based on the underlying taxonomy, would affect model performance. for example, lower-level knowledge which can be comprehended using basic cognitive skills (e.g., memorizing "what is the name of the girl?") is unlikely to have a big effect on solving tasks that require more advanced cognitive skills (e.g., making inference about "what would happen if the girl is taking a flight rather than a train?"). in addition to being accurate, being consistent and consequently predictable is critical for models to be applied by human users with confidence. using the proposed dataset and metrics, we conduct comprehensive evaluation of recent vision-language pre-training (vlp) models @xcite @xcite and a stateof-the-art multi-modal llm @xcite . we observe that for all three vlp models, the comprehension accuracy degrades for tasks requiring higher-level cognitive skills, in comparison to tasks requiring mere memorization of low-level details that can be directly identified from either textual or visual data. with zero-shot prompting experiments using gpt-4v (openai, 2023), we observe generally higher vqa accuracy over all levels. however, there is also an increasing tendency for the model to take shortcuts and bypass visual comprehension when considering tasks requiring higher-level comprehension, as shown in figure 2 . furthermore, via consistency analysis, we observe that model behaviors deviate from human comprehension patterns and intuitions at various scenarios. the results demonstrate the need for examining and improving current solutions based on theoretically-grounded criteria of reasoning and comprehension. in summary, our core contributions include the following: bloom's taxonomy first established by bloom et al. (1956) and further revised by anderson et al. (2001), bloom's taxonomy describes a framework categorizing learning objectives based on levels of complexity of underlying cognitive processes. there are six major categories, each represented by a set of skills and actions describing the corresponding cognitive processes. specifically, direct recall of concrete facts and knowledge is considered as the basis (level 1) of objectives requiring higher-order reasoning and abstraction, from understanding and being able to explain (level 2) to creating and generating original works (level 6). while bloom's taxonomy has been widely adopted in k-12 and college education as the guidance for designing teaching instructions and assessments @xcite , limited effort has been made to transfer the taxonomy to the machine learning domain as the guidance for analyzing, assessing and improving learning. @xcite propose a learningbased model for automatic classification of education questions based on bloom's taxonomy. @xcite use bloom's taxonomy to form proximal clarifying contexts and study their impact on language models. in this work, we propose a novel vqa dataset to support hierarchical model assessment guided by bloom's taxonomy. specifically, we use the action verbs associated with different bloom's levels to create templates for data collection. while it is noted in recent studies that using the action verbs alone may not be sufficient in fully describing the underlying learning process @xcite , the verbs provide us a concrete operational definition of the taxonomy as the guidance for understanding a coarse trend of model performance over data categorization. ## bloomvqa dataset data collection we propose a novel dataset for systematic assessment of multi-modal comprehension. we collect picture stories designed for educating young children from two creative commons (cc by 4.0) resources (storyweaver; book dash) which provide collections emphasizing an indian and an african cultural background respectively. we manually select 20 stories with relatively consistent artistic style, length (around 10-20 pages) and plot complexity so that tasks reflecting different levels of human comprehension and requiring different types of cognitive skills can be meaningfully collected based on the content of the stories. we filter out the stories with distorted artistic styles as they can pose additional challenges to both human annotators and learning models. we provide a web-based ui (details included in the appendix) where an annotator is asked to read through a picture story and then provide a set of 6 multiple-choice samples in english, each corresponding to one level in bloom's taxonomy. in bloom's taxonomy, each level of comprehension is associated with a set of action verbs describing underlying cognitive processes and skills (e.g., "identify" for level 1). we use the action verbs to construct a set of more than 70 question templates as our operational definition of bloom's taxonomy. for example, "how would you compare @xmath0. more examples are included in the appendix. for each sample, we collect one template-based question and 4 free-form answers including 1 correct answer and 3 incorrect answers from the same annotator. we collect data samples via amazon mechanical turk (amt) and manually review all inputs to select data with proper reflection of the story and the underlying bloom's level. for each story, we collect 10 sets of reviewed inputs from different annotators. to further reduce the bias in the dataset, we enforce annotators to provide answer choices with consistent lengths for the same question, with a length difference fewer than 5 words. detailed statistics about the data are shown in we perform human baseline on proposed dataset (1200 core vqa samples) with a small group of adult reviewers. the human baseline reports an average accuracy of 89% with 2% standard deviation. this preliminary study provides insights on the current gap between human and machine capabilities (gpt-4v reports an average accuracy of 75.3% as shown in story graph grounding vqa data to an established taxonomy provides not only theory-based categorization, but also guidance on systematic probing and generation of the data. based on bloom's taxonomy, we propose a hierarchical graph representation for picture story data. referred to as the story graph, this novel representation naturally extends the concept of scene graphs @xcite . while scene graphs focus on representing low-level information such as geometric relations between local entities based on individual images, in a story graph (figure 1 ), we represent different events in a single story as the nodes and represent event-level relations as the edges. in addition to low-level relations (e.g., temporal relations), we include a wide range of higher-level relations (e.g., logical relations, causal relations) corresponding to different levels of cognitive skills specified in bloom's taxonomy (e.g., making comparison, making inference). as shown in fig. 1 , each of our templates correspond to one type of the edges in the story graph describing a particular cognitive skill. the type of edge is the same for all instantiations of the template. for example, the template "<@xmath1, what would you do to change this action? <@xmath2, each encoding level 1 details such as the character involved and the action taken. in this way, rich and diverse knowledge about the story requiring different levels of comprehension is organized in a hierarchical manner. the benefits of the proposed story graph are multi-fold. by traversing through the underlying graph, we can achieve systematic augmentation of vqa data following the taxonomy encoded in the graph. for example, given a task about a base event, we can construct an augmented question by incorporating knowledge about each connected event as the context. this augmentation not only expands the dataset combinatorially, but also introduces novel metrics for assessing the consistency of learning models. as context information introduced in the augmentation is labeled with bloom's levels, consistency analysis on how context knowledge requiring different levels of comprehension would affect base tasks can be performed to characterize model reliability. ## experimental analysis to demonstrate the multi-faceted assessment enabled by our framework, we perform zero-shot experiments using three recent vlp models including clip @xcite , blip @xcite and blip2 @xcite with reported implementations @xcite . we further perform zero-shot prompting-based experiments using the state-of-the-art gpt-4v (openai, 2023). the baseline models are proposed to demonstrate a wide range of vision-language tasks probing multimodal comprehension. for clip experiments, we use pretrained vit-b-32 image transformer. for blip and blip2 experiments, we use pretrained models in the image-text matching (itm) mode @xcite . the models contain 583m and 188m trainable parameters as reported in the original papers. we use one nvidia tesla v100 gpu for experiments with vlp models. ## results and discussion multiple-choice accuracy in ## limitations constrained by the resources and availability of creative commons data, we collect a relatively small-scale dataset to serve as an evaluation set for characterization of model performance based on a theoretically-grounded taxonomy. while domain gaps may exist when evaluating models pre-trained with data from different domains, the focus of this work is not to optimize vqa accuracy in the picture story domain. instead, we focus on providing insights on model behaviors over different types of comprehension and measuring its consistency. our proposed hierarchical data representation and novel consistency metrics are generalizable and compatible with technologies such as prompting based data generation using llms which opens the opportunities for future research on scaling up the dataset. we collect picture story data from two resources emphasizing on indian and african backgrounds respectively. with such inputs, we encourage crossculture understanding. the utilization of stories designed specifically for young children further injects cultural neutrality, as the contents of the stories focus on core concepts appealing to children universally and have low cultural complexity. ## conclusions we present a novel dataset for systematic assessment of multi-modal comprehension engines. inspired by bloom's taxonomy from education research, we collect multiple-choice samples based on picture story data from creative commons resources. each sample is labeled with a cognitive skill required in solving the multiple-choice task and is associated with a specific level of comprehension from bloom's taxonomy. we further propose a novel hierarchical graph representation describing knowledge extracted from picture stories at different bloom's levels. the proposed story graph naturally extends the concept of scene graphs and maps data into the hierarchical taxonomy. we demonstrate that automatic data augmentation can be achieved by traversing through the underlying graph. with the proposed dataset, we can not only assess given models in a graded manner but also characterize the models with respect to consistency of their performance. the proposed data structure and metrics pave the way for a wide range of interesting future works, where hierarchical graph representation can be used to guide systematic data storage, retrieval and generation.