P roc VQA : Benchmarking the Effects of Structural Properties in Mined Process Visualizations on Vision–Language Model Performance (2025)

Abstract: Vision-Language Models have shown both impressive capabilities and notable failures in data visualization understanding tasks, but we have limited understanding on how specific properties within a visualization type affect model performance. We present ProcVQA, a benchmark designed to analyze how VLM performance can be affected by structure type and structural density of visualizations depicting frequent patterns mined from sequence data. ProcVQA consists of mined process visualizations spanning three structure types (linear sequences, tree, graph) with varying levels of structural density (quantified using the number of nodes and edges), with expert-validated QA pairs on these visualizations. We evaluate 21 proprietary and open-source models on the dataset on two major tasks: visual data extraction (VDE) and visual question answering (VQA) (with four categories of questions). Our analysis reveals three key findings. First, models exhibit steep performance drops on multi-hop reasoning, with question type and structure type impacting the degradation. Second, structural density strongly affects VDE performance: hallucinations and extraction errors increase with edge density, even in frontier models. Third, extraction accuracy does not necessarily translate into strong reasoning ability. By isolating structural factors through controlled visualization generation, ProcVQA enables precise identification of VLM limitations. ProcVQA is available at: https://github.com/kzintas/ProcVQA.

## introduction vision-language models (vlms) have shown both considerable promise and notable failures in interpreting data visualizations, a domain that requires cross-modal grounding between visual structures and symbolic encodings @xcite . while prior work suggests that models struggle with specialized charts such as gantt * equal contribution. or funnel charts over basic line or bar charts (han et al., 2023), and complex visualizations (e.g., more subplots) impair model performance (wang et al., 2024), there remains limited insight into which specific features within a visualization type influence model performance. one reason for this gap lies in the nature of existing benchmark datasets (tab. c.1), which are either synthetically generated (xia et al., 2024) or curated from diverse online sources (pew research center, 2025; owid, 2025; tableau public, 2025). although useful, these datasets generally lack systematic control over visualization features, making it difficult to rigorously assess their roles in shaping vlm performance. as one of the first attempts to address this gap, in this paper we focus on a particular type of visualization: node-link visualizations depicting mined process patterns (hereafter referred to as mined process visualizations). such visualizations show frequent patterns extracted from a set of event sequences, where the patterns highlight important events, workflows, and dependencies. they have been extensively studied in the areas of visual analytics and process mining and incorporated into commercial tools for various application domains such as clinical pathways @xcite we introduce procvqa, a benchmark of 118 mined process visualizations from eight real-world domains. the benchmark incorporates controlled variations on two visualization properties: structure type (three types of common node-link structures: linear sequences, trees, and graphs), and structure density (quantified by node and edge counts). using a standardized visualization generation pipeline (zinat et al., 2023), we control for encodings, typography, color, and glyphs, removing stylistic confounds that hinders analyzing structural effects in other benchmarks. this enables isolation of structural factors that affect vlm performance. we evaluate 21 vlms across two complementary tasks: visual data extraction (vde) and visual question answering (vqa). the vqa task is further categorized by reasoning complexity, with questions requiring either single-hop (value extraction, sequential reasoning) or multi-hop (value aggregation, unanswerable detection) reasoning (tab. a.3). this design allows us to analyze vlm performance across three dimensions: structure type, structural density, and reasoning complexity. our findings show that current vlms struggle with multi-hop reasoning, with performance varying by question type and visualization structure. extraction accuracy declines as structural density increases, with the degree of degradation differing across models. structure type also affects extraction accuracy, with most models performing weakest on graphs. to summarize, our contributions include: • procvqa, the first benchmark created for evaluating vlm performance on process visualizations across three structure types with varying structural densities. ## findings we evaluated 21 vlms on procvqa across visual question answering (vqa) and visual data extraction (vde), and present the key findings below. the evaluation metrics are described in app. b.2 and full quantitative results are available in app. e. ## conclusion procvqa is the first systematic benchmark for evaluating vision-language models on mined process visualizations. a central contribution of our work is the analysis of the influence of structural density, structure type, and task category, which uncovers degradation patterns obscured in traditional benchmarks. we find that edge density has a stronger effect on performance than node density. graphs are the most challenging structure type for most models, while trees are the most tractable. value extraction task is consistently manageable across all structures, while all models perform poorly on value aggregation task on graphs these findings demonstrate that vlm performance depends on the interplay between structural density, structure type, and reasoning complexity. we also observe that strong extraction capability may not translate to strong reasoning capability, and a wide variation in instruction-following ability, underscoring the need to consider extraction power, reasoning ability, and instruction adherence when deploying models (app. g.3). future work should focus on improving multihop reasoning, where models must perform a series of operations before reaching a solution. tasks such as tracing values across multiple paths and aggregating them (app. g.1) remain challenging even for frontier models.