Beyond End-to-End VLM s: Leveraging Intermediate Text Representations for Superior Flowchart Understanding (2025)

Abstract: Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) Limited controllability—users have minimal influence over the downstream task, as they can only modify input images, while the training of VLMs is often out of reach for most researchers. (ii) Lack of explainability—it is difficult to trace VLM errors to specific causes, such as failures in visual encoding or reasoning. We propose TextFlow, addressing aforementioned issues with two stages: (i) Vision Textualizer—which generates textual representations from flowchart images; and (ii) Textual Reasoner—which performs question-answering based on the text representations. TextFlow offers three key advantages: (i) users can select the type of text representations (e.g., Graphviz, Mermaid, PlantUML), or further convert them into executable graph object to call tools, enhancing performance and controllability; (ii) it improves explainability by helping to attribute errors more clearly to visual or textual processing components; and (iii) it promotes the modularization of the solution, such as allowing advanced LLMs to be used in the reasoner stage when VLMs underperform in end-to-end fashion. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance as well as its robustness. All code and data are publicly available.

## introduction flowcharts are extensively used to represent processes, algorithms, and workflows across a range of domains, including software engineering, business process modeling, and education. accurate interpretation of flowcharts is essential for tasks such as automation, decision-making, and analysis. given that flowcharts predominantly exist as images, the rise of large language models (llms) and large vision-language models (vlms) has led to the use of vlms for flowchart understanding in an end-to-end manner @xcite @xcite . while end-to-end vlms offer a straightforward approach to flowchart understanding, they exhibit two key limitations: (i) limited controllability-users have minimal capacity to improve performance, as they can only manipulate input images, while training vlms is resource-intensive and often inaccessible to most researchers. (ii) lack of explainability-it is difficult to trace errors in vlm outputs to specific failures, whether in visual encoding, reasoning, or other stages. to overcome these challenges, we introduce textflow, a framework that decomposes flowchart understanding into two stages: (i) vi-sion textualizer, which generates intermediate textual representations from flowchart images; and (ii) textual reasoner, which performs question-answering (qa) based on these text representations. this dual-stage framework provides three distinct advantages: (i) controllability-users can flexibly choose the type of text representation (e.g., graphviz, mermaid, or plantuml) and convert them into executable graph objects for enhanced performance; (ii) explainability-it improves error attribution by clarifying whether failures arise from visual or textual components; and (iii) modularity-the framework allows for the use of more advanced llms in the reasoner stage if vlms are inadequate for end-to-end flowchart understanding, restricting the vlm to the textualizer stage only. we evaluate textflow on the flowvqa @xcite and flowlearn @xcite datasets, combining various open-source and closed-source vlms/llms for the vision tex-tualizer and textual reasoner stages, respectively. our experiments demonstrate that us-ing claude-3.5 @xcite as the textu-alizer, alongside either claude-3.5 or gpt-4o (openai, 2024) as the reasoner, significantly outperforms end-to-end flowchart understanding using claude-3.5 or gpt-4o alone. further analysis reveals that (i) graphviz is generally the most effective textual representation for flowcharts; (ii) textflow exhibits robustness across different task categories, flowchart sources, orientations, and sizes; and (iii) the vision textualizer stage is more often the source of errors than the rea-soner stage, particularly when using claude-3.5 for both. our contributions are threefold: most of these works adopt end-to-end systems. for instance, @xcite proposed patch-trm, which employs a pyramid cross-modal transformer with diagram embeddings pre-trained on the icon dataset, while @xcite and @xcite explored various vlms for their respective tasks. ## methodology as illustrated in figure 1 , the approach textflow consists of two main stages: (1) vision textualizer: a vlm converts the visual components of the flowchart into a high-quality text representation. (2) textual reasoner: an llm or vlm based textural reasoner can use tools and interpret the text representations to answer questions related to the flowchart's logic and structure. by decoupling these tasks, textflow enables specialized models to focus on their respective strengths: vision textualizer process the visual elements, while textual reasoner handle reasoning tasks, thus improving accuracy and flexibility. next, we elaborate on vision textualizer (section 3.1) and textual reasoner (section 3.2), respectively. ## experiments dataset. we utilize the flowvqa dataset @xcite due to its extensive diversity in i) flowchart sources, including: code, wiki (stepby-step guides for daily tasks), and instruct (instructions for diy projects), and ii) task categories: flowvqa emphasizes spatial reasoning and understanding of decision-making flows within flowcharts, offering a robust resource to evaluate multimodal models' ability to interpret and process we accessed gpt-4o and claude3.5-sonnet via api, while all other open-source vlms/llms were used through hugging face with their official default settings. depending on the gpu memory requirements of each model, we utilized 1 to 4 nvidia a100(80gb) gpus. to ensure experimental accuracy and reproducibility, we employed a greedy decoding strategy (with @xmath0. in addition, to avoid the impact of image resolution on vqa performance, we used the highest resolution mode supported by each vlm. ## conclusion this work introduces textflow, a dualstage framework that leverages vlms/llms for flowchart understanding by breaking the process down into vision textualizer and textual reasoner. our experiments on the two benchmarks demonstrate its state-of-the-art performance, and detailed analysis reveals that graphviz is the most effective text representation for flowcharts. additionally, the system remains robust regardless of flowchart orientation and scale. beyond flowchart understanding, we believe that reasoning over intermediate text representations has the potential to generalize to other multimodal tasks, enhancing both usability and reasoning capabilities.