F low VQA : Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024)

Abstract: Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark’s potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.

## introduction since the inception of vision language models (vlms), tasks and benchmarks for visual question answering (vqa) and reasoning have received significant attention. most benchmarks evaluate pretrained model extraction capabilities, neglecting their ability to comprehend complex spatial relationships and visual logical reasoning. research on spacial path following or visual sequential reasoning in vlms is limited. current benchmarks for assessing vlm reasoning abilities mainly fall under vqa, a task formalized by @xcite , which involves generating responses to questions based on a given image. these works evaluate the spatial reasoning and visual information extraction abilities of vlms. visual grounding (vg) of a visual questionanswering system evaluates models' abilities to attribute their generations to different image regions referenced in the query @xcite . the absence of vg has been a frequent issue among the current vqa systems, manifesting in an overreliance on irrelevant parts of images or a complete disregard for the visual modality. existing benchmarks @xcite require models to rely on pre-trained knowledge to answer queries posed on image contexts. in this work, we aim to test the capabilities of vlms in following visual information without any pre-existing knowledge. to accomplish this, we delve into the realm of flowcharts, as depicted in figure 1 , which entail intricate structural configurations and path reconstruction, a considerably more challenging task compared to mere image comprehension. flowcharts emphasize sequential and logical reasoning, as they necessitate traversal of steps or decisions in a specific sequence. flowcharts are inherently visual, and provide a clear and structured method for representing processes, decision paths, and flows. unlike traditional text, which flows linearly, flowcharts require an understanding of directional logic; their flow is often multi-directional, representing various paths that can be taken based on certain conditions or decisions. despite being long and complex, flowcharts have compact, systematic representations and provide insights regarding information in a step-by-step manner. in this paper, we set out to answer a crucial question: "can modern vision language models effectively handle challenges that demand understanding both structural and semantic aspects, along with capturing macroscopic and granular context within visually complex yet straightforward flowcharts?" to tackle this question, we introduce flowvqa, a novel benchmark comprising intricate structural and path-based questions posed on lengthy flowchart images. we propose a novel approach to visual question answering (vqa) on flowchart tailored for vlm, with a focus on harnessing flowcharts as the primary contextual framework for visual logic and spatial reasoning. the flowvqa dataset, along with modeling and evaluation scripts, generation pipeline and prompts, and the human verification platform, can be accessed at https://flowvqa.github.io/ . ## proposed flowvqa resource in this section, we will see the details of the construction of flowvqa. we outline the process of collection of raw data from wild sources, multistep generation of mermaid scripts and flowchart images and complex q/a creation. ## experimental evaluation we address the following research questions through our experiments: rq1. does the introduced visual multimodal dataset present a significant challenge to current multimodal language learning models (vlms), and can it provide valuable insights that could contribute to their future advancement? rq2. is the efficacy of vlms influenced by factors such as (a) the source of flowcharts, (b) the type of questions posed, and (c) the level of complexity inherent in the flowcharts? ## related work vision language models have made large progress in diverse vision-language applications @xcite @xcite with multiple benchmarks being proposed to aid effective evaluation of visual and textual grounding capabilities of these models. the mmmu benchmark @xcite is designed to assess the model's inherent "subject-specific" knowledge and reasoning abilities across various subjects (such as technology, humanities, health, and more). benchmarks like textvqa and docvqa @xcite @xcite @xcite evaluate the models' fine-grained transcription abilities on low-resolution images. more complex multimodal reasoning tasks, such as mathvista @xcite , examine the models' abilities to integrate visual and mathematical logic. benchmarks focusing on spatial multimodal reasoning include chartqa @xcite @xcite and info-graphicvqa @xcite . chartqa is aimed at evaluating straightforward chart understanding and analysis, while infographicqa poses direct logical questions about data visualizations and charts. prior flowchart works. to our knowledge, there exists a study on flowchart qa @xcite , that suffers from major limitations. (i) synthetically generated flowcharts with randomized scripts, (ii) primarily poses structural questions and (iii) uses multiple choice-based questions to evaluate weaker existing models. other research in this domain addresses issues like flowchart object recognition and flowchart to code/script conversion, where a modest parallel flowchart resource is paired with corresponding code or script @xcite @xcite . however, notable limitations here include poor flowchart image quality, niche or overly complex context, structural imbalance (only linear or excessively complex), lack of ground truth scripts for flowcharts, and insufficient context for effective q/a or practical tasks. in contrast to these works, we construct a complex benchmark suitable to test practical applicability of existing vlms. the complex and diverse qa types ensure an effective and just evaluation over multi-modal visual and textual understanding. ## conclusion and future work in conclusion, this study evaluates the effectiveness of existing multimodal large language models (vlms) in reasoning upon a complex visual, sequential logical reasoning based task, flowvqa. we introduce the novel dataset resource, flowvqa, consisting of 2,272 flowchart images, mermaid.js scripts, 22,413 q/a pairs with gold standard answers. our extensive evaluation on these models with multiple strategies and scenarios highlights the need for advancements in architecture and prompting strategies in existing vlms. we also study the presence of any directional bias in the flowcharts by re-evaluating the test sets with an inverted flowchart subset. we find that both proprietary and open-source models suffer from directional bias due to lack of visual grounding and complex structural reasoning required for flowchart reasoning. future work. our work and resources give rise to many research avenues in (a) flowchart reasoning: flowvqa can be used to enhance the visual logic and reasoning capabilities of the models. constructing vlms that are flowchart specific is also a encouraging research direction. (b) graph-encoder models: in this study, we consider the graph nature of flowcharts solely to generate topological questions. this consideration can also be taken into account while designing model architectures and inference strategies to enhance structural reasoning in the base models.