Picturing Ambiguity: A Visual Twist on the W inograd Schema Challenge (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. However, applying this reasoning to multimodal domains, where understanding text and images together is essential, remains a substantial challenge. To address this, we introduce WinoVis, a novel dataset specifically designed to probe text-to-image models on pronoun disambiguation within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel evaluation framework that isolates the models’ ability in pronoun disambiguation from other visual processing challenges. Evaluation of successive model versions reveals that, despite incremental advancements, Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally surpassing random guessing. Further error analysis identifies important areas for future research aimed at advancing text-to-image models in their ability to interpret and interact with the complex visual world.

## introduction the interpretation of ambiguous constructs in language is crucial for assessing common-sense reasoning, with the winograd schema challenge (wsc) @xcite significantly influencing the evaluation of natural language understanding models. advances in transformer-based architectures have led large language models (llms) to achieve impressive results on wsc-based tasks, approaching nearhuman performance @xcite @xcite . ## constructing winovis in this section, we detail the methodology behind the creation of winovis, a novel dataset engineered to assess the pronoun disambiguation capabilities of text-to-image models. the integration of @xcite into our dataset generation workflow allowed for significant streamlining of the creation process, achieving reductions in both cost and time, enhancing reproducibility, and reducing the incidence of human error. our corpus construction cycle consists of two main stages: 1) the gpt prompt cycle; and 2) the manual filter process. a full visualization of the process is provided in appendix figure 10 . ## evaluating pronoun disambiguation in winovis this section outlines our systematic pipeline to evaluate the capability of stable diffusion models to accurately disambiguate pronouns within the context of winovis. our pipeline comprises four stages: 1) filtering out captioned images to remove visual representations that include embedded text; 2) enhancing the clarity of distributed attention attribution maps through noise reduction; 3) excluding images with significant heatmap overlap between referent entities from our analysis; and 4) determining the model's final pronoun association by establishing a decision boundary. ## error analysis in this section, we further examine the performance of the most effective model iteration, sd 2.0. we compare the results across our dataset categories outlined in section 3.2, namely disparate and distinct entities. the proportions of correct, incorrect, overlapped, neither, and captioned instances for both categories are visualized in figure 8 . ## related work multimodal reasoning the recent surge in popularity of generative models has underscored the necessity for explainable creativity @xcite , leading to a significant body of research investigating the determinants of high-quality prompts for image generation @xcite @xcite . despite these advancements, the evaluation of how vision models actually interpret prompts is largely underexplored. most studies focus on the models' semantic understanding of terms @xcite @xcite or susceptibility to bias @xcite . these evaluations often involve direct, unambiguous prompts, sidestepping more nuanced challenges. winovis addresses this gap by evaluating the common-sense reasoning of models through the lens of pronoun resolution. this challenge not only expands the scope of assessment for generative models but also sets a new benchmark for understanding their capabilities in interpreting complex linguistic structures. ## conclusion this paper presented winovis, a new approach to test how well text-to-image models like stable diffusion handle pronoun disambiguation. our work reveals significant gaps in these models' abilities to interpret ambiguous scenarios accurately. central to our contribution is a novel evaluation framework designed to isolate common-sense reasoning in pronoun disambiguation from well-studied challenges such as typographic attacks and semantic entanglement. future research should build on our groundwork to develop models that not only generate visually compelling images but also accurately understand the narratives and relationships within them.