F low2 C ode: Evaluating Large Language Models for Flowchart-based Code Generation Capability (2025)

Abstract: While large language models (LLMs) show promise in code generation, existing benchmarks neglect the flowchart-based code generation. To promote further research on flowchart-based code generation, this work presents Flow2Code, a novel benchmark for flowchart-based code generation evaluation. The evaluation dataset spans 15 programming languages and includes 5,622 code segments paired with 16,866 flowcharts of three types: code, UML, and pseudocode. Extensive experiments with 13 multimodal LLMs reveal that current LLMs can not generate code based on flowcharts perfectly. Besides, experiment results show that the supervised fine-tuning technique contributes greatly to the models’ performance. The dataset will be publicly available.

## introduction the code generation task aims to convert specific requirements into executable code @xcite , which attracts interest and focus from the academic and industrial communities. recently, for automatic code generation, large language models (llms) @xcite @xcite @xcite exhibit substantial potential and show alluring application value for enhancing productivity, minimizing human error. to comprehensively evaluate and understand the code generation capabilities of emerging llms, substantial efforts have been devoted to establishing and refining code generation benchmarks. specifically, as shown in figure 1 , current code generation benchmarks can be classified into two categories: those based on textual descriptions, such as humaneval @xcite and mbpp @xcite , and those based on images of programming problems or matplotlib plots, such as * corresponding authors: zeming liu, xiaoming shi. mmcode @xcite and plot2code @xcite . despite the potential value, these works fundamentally suffer from a critical flaw: the lack of flowchart-based code generation evaluation. compared to textual descriptions, images of programming problems, or matplotlib plots, flowcharts offer a more effective and intuitive way to understand and visualize program logic such as decisions, loops, and conditionals, making them accessible to both programmers and non-programmers alike @xcite . flowcharts mainly consist of three types: basic code flowcharts, unified modeling language (uml) flowcharts, and pseudocode flowcharts @xcite . specifically, basic code flowcharts represent the step-by-step execution of a program, uml flowcharts are a formal graphical representation of an object-oriented system's structure, and pseudocode flowcharts are a high-level abstraction of program logic and are represented by natural language. despite these advantages, there is a notable gap in code generation benchmarks, with no dedicated datasets or frameworks for generating code from flowcharts. this limitation hinders the ability of llms to fully utilize flowcharts for code generation. to address this gap, this work first introduces flow2code, a comprehensive code generation benchmark that includes three types of flowcharts and corresponding code in 15 programming languages. the construction of flow2code consists of three parts: the creation of code and uml flowcharts, pseudocode conversion, and data checking. after the construction of the dataset, a twostep human evaluation process is employed to ensure data quality, which includes both code verification and validation of pseudocode flowchart transformations. finally, the dataset is obtained with a total of 5,622 code segments, 16,866 flowcharts and includes 15 programming languages, offering a rich resource for evaluating code generation tasks. to further conduct the evaluation of current datasets code flowchart uml flowchart pseudocode flowchart multimodal multilingual samples apps @xcite ✗ ✗ ✗ ✗(text) ✗ 10,000 humaneval @xcite ✗ ✗ ✗ ✗(text) ✗ 164 mbpp @xcite ✗ ✗ ✗ ✗(text) ✗ 974 ds-1000 @xcite ✗ ✗ ✗ ✗(text) ✗ 1,000 codecontests @xcite ✗ ✗ ✗ ✗(text) ✓( 3 ) 13,610 mbxp @xcite ✗ ✗ ✗ ✗(text) ✓( 13 ) 12,425 classeval @xcite ✗ ✗ ✗ ✗(text) ✗ 100 codereval @xcite ✗ ✗ ✗ ✗(text) ✓(2) 460 humaneval-x @xcite ✗ ✗ ✗ ✗(text) ✓(5) 820 mceval @xcite this work makes the following contributions: ## related work this section first reviews the current research on code generation benchmarks (section 2.1), followed by an overview of research on multimodal benchmarks (section 2.2). ## dataset construction this section first describes the data source and selection for the flow2code dataset (section 3.1), followed by the flowchart construction process (section 3.2), and concludes with an analysis of the dataset (section 3.3). ## experiments and results this section first presents the experimental setting (section 4.1) and evaluation metrics (section 4.2), followed by the model baselines (section 4.3) and experimental results (section 4.4). ## conclusion we introduce flow2code, a multimodal dataset combining flowcharts and code in 15 programming languages, designed to advance research in code generation. the dataset includes three flowchart types-code, uml, and pseudocode-providing a rich resource for training and evaluating mllms. we propose a new benchmark, flow2code, to assess mllms on code generation from flowchart inputs. extensive evaluations of 13 mllms highlight their strengths and weaknesses, providing valuable insights for future research.