Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents (2025)

Abstract: Automated service agents require well-structured workflows to deliver consistent and accurate responses to customer queries. However, such workflows are often undocumented, and their automatic extraction from conversations remains largely unexplored. In this work, we present a novel framework for extracting and evaluating dialog workflows from historical interactions. Our extraction process involves two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation step using question-answer-based chain-of-thought (QA-CoT) prompting. To comprehensively evaluate the quality of the extracted workflows, we introduce an automated simulation framework with agent and customer bots that measures their effectiveness in resolving customer issues. Extensive experiments on the ABCD and SynthABCD datasets show that our QA-CoT technique improves workflow extraction by 12.16% in average macro accuracy over the baseline. Moreover, our evaluation method closely aligns with human assessments, offering a reliable and scalable framework for future research.

## introduction llms are transforming customer service by enabling automated bots to respond accurately and consistently to customer inquiries. these bots rely on well-structured workflows to guide interactions, ensure efficient issue resolution, and maintain adherence to company policies @xcite . however, high-quality workflows are scarce, making it difficult to ensure agent consistency and quality. traditional workflow creation methods typically depend on manual efforts @xcite @xcite @xcite , which are time-consuming, prone to becom-figure 1 : an example showing the derivation of a workflow from historical conversations. full workflow is shown in figures 3 and 5 . in this paper, we propose a novel two-step pipeline for automatically extracting dialog workflows from customer-agent interactions. these conversations often contain noise, such as agents combining multiple steps or sequencing them incorrectly. moreover, customer interactions typically cover only a subset of the established rules, making it difficult to capture a complete workflow. to address these challenges, our pipeline first enhances the retrieval process by identifying key procedural components (figure 2 )-including intents, slot values, and resolution steps-and leveraging them to select the most relevant conversations while filtering out non-compliant ones. in the second step, we introduce a structured qa-based chain-of-thought (qa-cot) prompting "intent": "customer wants to refund an order before shipping.", "slot_values": { "full_name": "chloe zhang", "username": "czhang94", "email": "czhang94@email.com", "order_id": "1553732700", "refund_method": "original credit card", "shipping_status": "not shipped" }, "resolution_steps": [ "agent asked for the customer's full name.", "agent requested the username, email, and order id.", "agent asked how the customer would like the refund processed.", "agent inquired about the account id.", "agent asked about the shipping status of the order." ] figure 2 : an example of procedural elements extracted from a conversation by the gpt-4o-mini llm. another key contribution of our work is a robust multi-step evaluation pipeline for dialog workflows, featuring an end-to-end methodology that closely aligns with human assessments. we simulate interactions between a service agent bot (equipped with the predicted workflow and relevant system data) and a customer bot (provided with the intent and user-specific details). after the conversation concludes, we evaluate task success by comparing the outcome produced using the predicted workflow with the expected outcome derived from the ground truth workflow (see figure 5b for an example). we evaluate our proposed qa-cot method against standard prompting strategies using the abcd @xcite dataset and an llmgenerated variant, synthabcd. while abcd originally grounds conversations in predefined dialog workflows, we repurpose it for the task of extracting workflows from conversations. to eliminate human-induced errors present in abcd conversations, we generate synthabcd using llms, ensuring error-free dialogues that adhere strictly to the same workflows. this enables controlled and reliable evaluation under an idealized setting. our qa-cot method outperforms other prompting strategies across multiple llms, including gpt-4o, opus, sonnet, and gemini. for example, on the abcd dataset, it achieves an 11.81% improvement over the baseline with gpt-4o. moreover, incorporating explicit qa-cot reasoning enhances the performance of state-of-the-art reasoning models such as openai o1, o1-mini, o3-mini, and deepseek-r1 (guo et al., 2025) across both datasets. ## related work dialog workflows are a specialized form of procedural knowledge. while workflow extraction has received little attention, automatic procedural extraction has been widely studied, primarily focusing on "how-to" documents @xcite @xcite and instructional videos @xcite @xcite . these works typically model linear sequences of explicitly stated actions, aiming to either predict procedural steps or generate summaries of task execution @xcite . in contrast, our work tackles dialog workflows, where actions are often implicit and depend on previous steps, user inputs, and system responses. this introduces decision-dependent variability, making extraction significantly more challenging than predicting fixed procedural sequences. specific to dialog systems, there has been extensive research on studying structures in taskoriented dialogs @xcite @xcite @xcite @xcite @xcite @xcite @xcite , focusing on how dialogs evolve using dialog acts, intent-slot pairs, or turnlevel dependencies. our work parallels workflow discovery @xcite @xcite , which aims to predict the optimal next dialog action from the conversation's current state and all available actions. however, unlike this, we focus on extracting global workflows applicable across conversations for a specific intent. this increases the complexity of the task, as the model must filter out noisy actions, and consolidate multiple potential actions sequences from different conversations. additionally, we also propose a new implementer: the agent must ask for the username, email address, and order id to validate the purchase. guide: how does the customer's membership level impact the return process? implementer: the customer's membership level determines the return policy duration: ## dialog workflow extraction the dialog workflow extraction task aims to identify and organize all essential steps required to resolve customer issues across various scenarios into structured guidelines. for example, figure 3a illustrates a workflow for handling customer complaints about a bill they never purchased, while figure 5a shows the corresponding state machine diagram, highlighting different sub-flows. these sub-flows account for scenarios such as assisting gold, silver, guest, or bronze members; addressing cases with or without system errors; and accommodating customers who provide either their full name or account details. in this example, the goal is to generate a workflow that comprehensively captures all variables, including membership levels, the pres-ence of system errors, and the type of customer information available. we focus on two key aspects to improve dialog workflows extraction: retrieval, which involves selecting the most relevant historical conversations for the task, and extraction, which uses a structured qa-cot prompting approach to derive workflows from the selected conversations. ## dialog workflow evaluation evaluating workflow requires careful attention to the relationships and dependencies between constituent steps to ensure the predicted workflow aligns with the expected decision-making process. we encountered significant challenges in performing this evaluation at the workflow level, primarily due to the complex dependencies between steps that are difficult to assess consistently. in an initial study, we asked human annotators to evaluate whether each step in the ground-truth workflows was accurately represented in the predicted workflows, using a set of 10 llm-extracted workflows. the results showed moderate inter-annotator agreement, with a cohen's κ score of 0.45, highlighting the difficulties even humans face in identifying nuanced procedural differences. to address these challenges, we propose an endto-end (e2e, figure 5 ) evaluation framework that works in five stages foot_0 : we evaluated the accuracy of our e2e simulation pipeline and found that the overall accuracy across steps 1 to 4 was 94.81%. moreover, step 5 exhibited strong alignment with human evaluation, achieving a cohen's kappa score of 0.92. these results suggest that our e2e evaluation closely mirrors human judgments (for details, see appendix b). ## error analysis we manually reviewed all eight qa-cot systems to identify cases where they failed to handle all scenarios correctly (example in figure 8 ). one common mistake was confusing system-available information with details that should be requested from the user. this issue was particularly frequent in shipping-related tasks, where models incorrectly assumed they should check the shipping status in the system instead of asking the customer. o1 and deepseek-r1 made this mistake in five intents, while o3-mini was the only model that avoided it. this error did not stem from the conversation data but rather from the llm's parametric knowledge, as it appeared in both the abcd and synthabcd datasets. another common issue was models tend to ignore alternative options. instead of allowing a choice, they often predicted only one option. for example, when a workflow required asking for either the "full name or the account id", models typically picked just one (e.g., ask "full name") instead of keeping both possibilities. a third major issue was failing to follow conditional logic at branching points. for example, a workflow might require asking for the membership level only if there is no system error. however, predicted workflows often ignored this condition and included a fixed sequence, combining steps incorrectly instead of following the intended logic. we observed two additional patterns that did not affect accuracy but impacted efficiency. first, models often merged multiple steps into a single step, particularly during the early steps in workflows. for example, a workflow might specify sequential steps such as verifying the account id or full name, followed by email id, order id, and username. rather than maintaining these as distinct steps, models frequently combined them, thereby reducing the average number of utterances required to resolve an issue. second, when the original workflow allowed multiple conditions to proceed, such as purchase date within 90 days, original receipt, or original packaging, models tended to convert this flexibility into a strict sequence. for instance, instead of allowing any of the conditions to be sufficient, models often predicted checking the purchase date first, then asking about the receipt if the date exceeded 90 days, and so on. although these behaviors do not reduce accuracy, they negatively impact efficiency by increasing the number of utterances compared to the ground-truth workflow. ## conclusion we introduced a novel framework for extracting and evaluating dialog workflows from historical conversations. our proposed qa-cot prompting approach outperforms standard prompting strategies, yielding a 12.16% improvement in average macro accuracy. furthermore, the strong alignment between our end-to-end evaluation and human assessments highlights the robustness and reliability of our evaluation framework. we believe this work provides a robust foundation for structured workflow extraction and evaluation, and will encourage future research to develop more effective methods.