S truct F low B ench: A Structured Flow Benchmark for Multi-turn Instruction Following (2025)

Abstract: Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. These structural dependencies not only reflect user intent but also establish an essential second dimension for the instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark defines an innovative structural flow framework with six fundamental inter-turn relationships. These relationships introduce novel structural constraints for model evaluation and also serve as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models’ comprehension of multi-turn dialogue structures. The code is available at https://github.com/MLGroupJLU/StructFlowBench.

## introduction the rapid advancement of large language models (llms) in multi-turn dialogue systems has elevated instruction-following capabilities to a pivotal research frontier in human-ai interaction @xcite . current evaluation methodologies bifurcate into two streams: multi-turn dialogue evaluations focusing on general capabilities @xcite @xcite and instruction-following analyses emphasizing finegrained constraint compliance @xcite figure 1 : the structural flow taxonomy includes six fundamental structures, each used to describe the inter-turn relationships in multi-turn dialogues. it can be applied to analyze any dialogue and generate specific structural flows. @xcite . more recent research has started to model the composition of intra-turn constraints @xcite . however, current evaluation methodologies treat multi-turn dialogues merely as simple concatenations of single-turn interactions, neglecting users' planning and intentionality in extended conversations. this leads to three critical limitations: (1) failure to model complex scenarios: multi-turn dialogue data constructed with simplistic linear thinking cannot accurately capture key characteristics of real-world complex conversations, such as logical coherence, user goal clarity, and natural transitions. (2) methodological bias: single-turn evaluation strategies fragment inter-turn structural connections, overlooking multi-turn structural constraints. (3) analytical deficiency: existing approaches overemphasize intra-turn constraint compliance while lacking a systematic framework to characterize dialogue structural flow. to bridge these gaps, we introduce structflow-bench, a novel instruction-following benchmark integrating a multi-turn structural flow framework. it consists of two key components: 1) dualconstraint evaluation system, comprising two layers-intra-turn constraint evaluation and interturn constraint evaluation-offers a more comprehensive assessment of llms' multi-turn dialogue instruction-following capabilities. building on existing intra-turn constraint assessments, we integrate five newly proposed structural constraints with eight intra-turn instruction constraints to capture the relationships between dialogue turns. these structural constraints account for inter-turn dependencies, ensuring that models are evaluated not only on their ability to satisfy individual constraints but also on their capacity to maintain logical coherence across multiple turns. 2) six-category structural flow taxonomy, encompassing six fundamental inter-turn relationships: follow-up, refinement, recall, summary, expansion, unrelatedness. figure 1 illustrates the structural flow taxonomy, which is defined in detail in section 3.1. this taxonomy serves a tripartite function: (a) structural diagnosis: it enables a structured analysis of cross-turn structural rationality, helping to identify inconsistencies in dialogue flow and ensuring that model responses align with the expected discourse structure. figure 2 the taxonomy provides configurable structural parameters that guide task-specific dialogue simulation, allowing for the tailored generation of multi-turn conversations with predefined structural patterns. this not only enhances dataset diversity but also supports the development of more robust instruction-following models adaptable to varied real-world applications. we summarize our contributions as follows: ## structflowbench this section introduces the structural flow framework and constraint categories, details the data construction pipeline and benchmark statistics, and outlines the evaluation protocol. ## conclusion in this work, we address key limitations in current multi-turn instruction-following research by introducing structflowbench, a novel benchmark designed to capture the structural intricacies of complex dialogue scenarios. by incorporating a dualconstraint evaluation system and a six-category structural flow taxonomy, we provide a more comprehensive framework for assessing the logical coherence, goal clarity, and transition naturalness of multi-turn dialogues. our evaluations of 13 representative llms reveal critical insights into the structural processing capabilities of both closedsource and open-source models, offering valuable guidance for future advancements in instructionfollowing systems. through structflowbench, we lay the foundation for more robust, realistic, and contextually aware dialogue systems.