Synthetic Dialogue Dataset Generation using LLM Agents (2023)

Abstract: Linear programming (LP) problems are pervasive in real-life applications. However, despite their apparent simplicity, an untrained user may find it difficult to determine the linear model of their specific problem. We envisage the creation of a goal-oriented conversational agent that will engage in conversation with the user to elicit all information required so that a subsequent agent can generate the linear model. In this paper, we present an approach for the generation of sample dialogues that can be used to develop and train such a conversational agent. Using prompt engineering, we develop two agents that “talk” to each other, one acting as the conversational agent, and the other acting as the user. Using a set of text descriptions of linear problems from NL4Opt available to the user only, the agent and the user engage in conversation until the agent has retrieved all key information from the original problem description. We also propose an extrinsic evaluation of the dialogues by assessing how well the summaries generated by the dialogues match the original problem descriptions. We conduct human and automatic evaluations, including an evaluation approach that uses GPT-4 to mimic the human evaluation metrics. The evaluation results show an overall good quality of the dialogues, though research is still needed to improve the quality of the GPT-4 evaluation metrics. The resulting dialogues, including the human annotations of a subset, are available to the research community. The conversational agent used for the generation of the dialogues can be used as a baseline.

## introduction linear programming (lp) is a mathematical optimization technique widely employed to address a multitude of real-world challenges, ranging from resource allocation in supply chain management to portfolio optimization in finance. despite the ubiquity of lp problems and their practical importance, many individuals, particularly those without specialized mathematical backgrounds, often struggle to formulate the appropriate linear models for their specific problem instances. this barrier hinders the broader utilization of lp techniques, especially among non-experts. to mitigate this challenge, we propose the development of a goal-oriented conversational agent capable of assisting users in constructing accurate linear models for their unique problem scenarios. this conversational agent would engage users in a dialogue, eliciting relevant information pertaining to the problem, and subsequently generate the corresponding linear model. this paper focuses on an essential aspect of creating such an agent -the generation of synthetic dialogues that can be employed to train and evaluate the conversational agent's performance. our methodology leverages prompt engineering to construct two distinct agents: one simulating the conversational agent's behavior, and the other emulating the user's responses during problem-solving interactions. the agents are designed to engage in purposeful dialogues aimed at extracting the necessary information from the user to construct a valid linear model. to facilitate this process, we utilize a set of text descriptions of linear problems, accessible only to the user agent, sourced from the nl4opt dataset @xcite . these text descriptions serve as the basis for the dialogues and enable the conversational agent to iteratively gather the critical information required for problem formulation. in addition to the generation process, we propose an extrinsic evaluation approach for assessing the quality and effectiveness of the generated dialogues. specifically, we evaluate how well the summaries generated by the dialogues align with the original problem descriptions from nl4opt. this evaluation encompasses both human assessments, where human judges evaluate the quality of dialogue summaries, and automated metrics to quantitatively measure the informativeness of the generated summaries. our preliminary results from human and automatic evaluations indicate that the generated dialogues exhibit a high degree of fidelity to the original problem descriptions, thereby demonstrating the quality of the synthetic dialogues generated. the contributions of this paper are: ## methodology this section outlines the methods employed to accomplish two objectives of this study: automating the generation of dialogue datasets, and evaluating the quality of the generated dialogues. we utilize a dual-agent setup leveraging llm, in our case, openai's @xcite , to simulate a conversation between a user and an assistant focusing on linear programming problems. ## evaluation strategy to evaluate the quality of the generated dialogues, we conduct an extrinsic evaluation. the evaluation process focuses on an underlying hypothesis that serves as the core of our study: a correct summary generated by the qg agent would imply that the agent has posed the appropriate and necessary questions to restore the original problem statement effectively. the hypothesis is grounded in several key assumptions and logical deductions: qg agent: hello there! i'm optimouse, your friendly helper. i'm here to assist you in making the best decision for your situation. could you please tell me what you're trying to achieve or improve? qa agent: hello! i'm looking to maximize profit for my furniture store while considering space, capital, and product ratio constraints. qg agent: that sounds like a great goal! to help you better, could you please tell me what types of furniture you sell in your store? qa agent: we sell dining tables and chairs in our store. qg agent: that's great! now, could you please tell me how much profit you make from selling a dining table and how much from selling a chair? qa agent: we make a profit of @xmath075 from selling a chair. qg agent: that's helpful to know. now, could you please tell me how much space a dining table and a chair take up in your store? qa agent: a dining table requires 8 sq ft of floor space while a chair requires 2 sq ft. ... ... ... ... qg agent: that's a very important detail. now, could you please tell me if there are any other constraints or requirements that we need to consider for your furniture store? qa agent: no, there are no other constraints or requirements for my furniture store. qg agent: great! let me summarize the information: -you sell dining tables and chairs in your furniture store -you make a profit of @xmath175 from selling a chair. -a dining table requires 8 sq ft of floor space while a chair requires 2 sq ft. -you have 500 sq ft of space available in your store. -you have a maximum of @xmath21000 to purchase and a chair costs @xmath3350 and the profit per chair is @xmath41000 in capital and a chair ties up @xmath520000 worth of capital tied up at any time. formulate an lp to maximize profit. extrinsic evaluation. in dialogue systems, particularly goal-oriented systems, the dialogue often progresses through a series of questions and answers to reach a particular end state @xcite . therefore, the questions asked directly impact the quality and accuracy of the final output -here, the summary of the original problem statement. we, therefore, conduct an extrinsic evaluation of the dialogues. ## summary and conclusion this paper presents a dataset for the task of eliciting information from the user through a dialogue with a conversation agent. the specific use of the information elicited is for automatic modeling of linear optimization problems. this is per se a very useful task with broad potential applications, but the methods for data generation and evaluation proposed here can be adopted easily for other possible tasks. the data and human evaluations are available to the research community. 1 the dialogue was generated in a dual-agent llm setup where a question generation agent acted as the machine agent who elicited information, and a question answering agent acted as the human who had the information about the problem to model. the question generation agent can be used as a baseline agent. the human evaluation results indicate that this baseline may be effective for the task. the dialogues for a subset of 28 lp problems were evaluated using an extrinsic evaluation that judged whether summaries generated by the dialogues matched the key information from the original problem descriptions. the evaluation was conducted by human evaluators and automatically. among the automatic evaluation metrics, besides well-known automatic metrics rouge and bertscore, we designed another gpt-4 agent that mimicked the human evaluators. the results indicate a reasonable correlation between rouge l, bertscore p, and the average human information precision scores, and this is slightly better than the correlation between the gpt4 agent and the human ip scores. as further work, we intend to refine the prompts used for the evaluation approach with gpt-4. in addition, we will conduct more exhaustive types of evaluation on the data set that might be more suitable to the specific domain of linear programming modeling. in particular, we plan to analyse the generated dialogues at the level of the dialogue turns.