C omm VQA : Situating Visual Question Answering in Communicative Contexts (2024)

Abstract: Current visual question answering (VQA) models tend to be trained and evaluated on image-question pairs in isolation. However, the questions people ask are dependent on their informational needs and prior knowledge about the image content. To evaluate how situating images within naturalistic contexts shapes visual questions, we introduce CommVQA, a VQA dataset consisting of images, image descriptions, real-world communicative scenarios where the image might appear (e.g., a travel website), and follow-up questions and answers conditioned on the scenario and description. CommVQA, which contains 1000 images and 8,949 question-answer pairs, poses a challenge for current models. Error analyses and a human-subjects study suggest that generated answers still contain high rates of hallucinations, fail to fittingly address unanswerable questions, and don’t suitably reflect contextual information.

## introduction visual question answering (vqa), the task of providing an answer to a question given an image, measures a model's ability to synthesize visual and textual modalities, and has many promising realworld applications. for example, when images online can't be seen and accessed, it severely affects people's abilities to educate themselves, socially engage, and stay informed @xcite @xcite , and vqa models are an opportunity for providing interactive accessibility to such visual content at scale @xcite . while most prior vqa datasets focus on investigating image-text alignment as a decontextualized task @xcite @xcite , we aim to reframe it as a human-centric communicative problem, moving it closer to a realworld interactive setting. according to a pragmatic bayesian view of communicative actions, people tend to ask questions that maximize the contextually relevant information gain based on their existing prior beliefs about the world @xcite , following fundamental pragmatic principles @xcite . based on these linguistic insights, we argue that prior vqa datasets largely do not consider two central communicative drives that limit their utility. the information people aim to obtain (and consequently the types of questions they ask) varies with (1) the person's information needs based on their goals when encountering the image, and (2) the person's prior knowledge of the image content. thus we introduce commvqa, a benchmark that treats vqa as an inherently communicative task. 1 to investigate people's information needs, com-mvqa consists not only of images, questions, and answers, but also of image descriptions and plausible communicative scenarios for each image. prior image accessibility research with blind and lowvision (blv) participants shows that the information people want from an image is dependent upon the scenario in which the image appears @xcite @xcite . for example, if a person encounters an image when they are shopping online, they are likely to ask questions about the brands within the image, while if they're browsing the news, the perceived purpose of the image shifts, and they are likely to ask questions about the event occurring within the image @xcite . we therefore define a scenario as the type of website (e.g., a shopping website) combined with a goal for viewing it (e.g., to buy a gift) and expect it to shape question elicitation ## related work with commvqa, we aim to situate the abstract vqa task within communicative settings. this builds on prior vqa dataset work (section 2.1), and communicative insights from linguistics and human-computer interaction (section 2.2) ## the commvqa dataset the commvqa dataset was constructed in five main steps, as visualized in figure 1 . first, we sourced images from wikipedia, and elicited both plausible scenarios and descriptions for the images from gpt-4v @xcite . to ensure description quality, we conducted a human-subject study to edit the descriptions. we then elicited questions and answers for the dataset from usbased crowdworkers on prolific. all human-subject studies were conducted with irb approval. ## dataset analysis the final commvqa dataset consists of 8,949 unique question-answer pairs, spanning across 2000 unique image-scenario pairs. to better understand the challenges posed by this dataset, we now provide an analysis of commvqa. examples from the dataset, sampled to cover all scenarios, are given in appendix figure 5 . we additionally conducted an analysis of a subset of the data and found similar patterns in model responses on this smaller dataset, suggesting our data is sufficiently large to obtain generalizable patterns. for more details, see appendix c. ## model benchmarking in this section, we investigate the performance of four state-of-the-art vision-language models on commvqa and to what extent providing context to the models improves their performance. we selected four models to benchmark on this dataset: llava @xcite , blip-2 @xcite , mplug-owl (ye et al., 2023), and idefics @xcite . to maximally enable the reproducibility of this work and support the public development of models, we focus on open-source models. llava is trained on a multimodal instruction-following dataset, and exhibits strong performance across several vqa datasets @xcite . mplug-owl @xcite is also instruction-tuned, and displays competitive performance on various vqa benchmarks, including vqa-v2 @xcite . idefics @xcite is an open-access reproduction of flamingo @xcite , and was trained on a naturalistic web-scale dataset of interleaved image-text documents, including 141 million web pages. we selected llava, mplug-owl, and idefics since their instruction-tuned nature allows us to straightforwardly integrate context. we also included a non-instruction-tuned model, blip-2, a general-purpose vision-language model that is additionally fine-tuned on the vqa task. ## conclusion visual question answering models are a promising tool for making visual content accessible to all. with commvqa, we move the contextually isolated vqa task into a communicative setting that starts reflecting the diversity of downstream use, while keeping close control over the nature of the dataset to ease interpretability. we find strong evidence from dataset analysis that the types of relevant questions and answers change with the contextual domains where images appear. we also find evidence that integrating contextually relevant information improves model performance. our results suggest that the path towards building viable vqa systems requires a focus on the wider communicative context where images appear.