M ono TOD ia: Translating Monologue Requests to Task-Oriented Dialogues (2025)

Abstract: Data scarcity is one of the main problems when it comes to real-world applications of transformer-based models.This is especially evident for task-oriented dialogue (TOD) systems, which require specialized datasets, that are usually not readily available. This can hinder companies from adding TOD systems to their services.This study therefore investigates a novel approach to sourcing annotated dialogues from existing German monologue material.Focusing on a real-world example, we investigate whether these monologues can be transformed into dialogue formats suitable for training TOD systems.We show the approach with the concrete example of a company specializing in travel bookings via e-mail. We fine-tune state-of-the-art Large Language Models for the task of rewriting e-mails as dialogues and annotating them.To ensure the quality and validity of the generated data, we employ crowd workers to evaluate the dialogues across multiple criteria and to provide gold-standard annotations for the test dataset.We further evaluate the usefulness of the dialogues for training TOD systems.Our evaluation shows that the dialogues and annotations are of high quality and can serve as a valuable starting point for training TOD systems.Finally, we make the annotated dataset publicly available to foster future research.

## introduction the rise of large language models (llms) has inspired many new fields of research and applications. one of the factors enabling their success is their capability to follow natural language prompts @xcite , increasing and simplifying control over the model's output. in general, chatbots can be roughly categorized into task-oriented dialogue (tod) systems and 1 https://github.com/sebastian-steindl/ monotodia open-domain dialogue systems @xcite . tod systems can be seen as a natural language interface to one (or multiple) external services, helping the users to achieve a certain task. these external services can often be treated as a database or an endpoint that is being queried. the request will be constructed in predefined slots that are being filled by the tod system during the conversation. everyday examples include actions like booking a restaurant or a train ticket. furthermore, multiple domains can be combined within one dialogue, enabling the user to, e.g., book a complete vacation, including flights, hotel, and restaurants, within one conversation. for real-world productive use, the requests to the services will usually need to be made on live data, e.g., to get current prices and availabilities. the tod system has to complete three subtasks to fill slots and build requests to the external services: understanding the user (natural language understanding, nlu), deciding on how to react (policy planning, pp), and finally creating a response (natural language generation, nlg) @xcite . compared to open-domain dialogues, tods are usually multi-turn but short, constrained to certain domains, and highly structured @xcite . while tod systems were traditionally rule-based, current approaches use deep learning and transformers @xcite @xcite , achieving better results but requiring large amounts of training data. this work thus studies whether the current advances in llms can make training a tod system more accessible by translating existing monologue data into annotated, task-oriented dialogues. we fine-tune a state-of-the-art llm to automatically translate the monologues into dialogues. in a second step, they are annotated with a llm. the method is demonstrated on real-world e-mail requests. to assess the quality of the resulting dialogues and annotations, we perform human evaluation and investigate the usefulness of the data for training of downstream tod systems. our results indicate that style translation with llms could be a viable approach to cold start and low-resource problems for tod systems. we publish the resulting dataset with gold-standard annotations for the test split. an example of an e-mail and the resulting dialogue is shown in fig. 2 . further examples of dialogues generated with monotodia are shown in figures 6 and 7 in appendix g. ## problem statement the need for training data is aggravated by the special requirements for the data in tod systems, making data collection tedious, expensive and thus a fundamental bottleneck for the development of tod systems @xcite @xcite . in collaboration with a german enterprise, we thus investigate an approach to tackle this problem: translating existing nondialogue data to multi-turn tods. we showcase this on e-mails, which can be seen as monologue requests in this scenario. this would drastically reduce the data collection and labeling effort while staying close to real-world, domain-specific data and tackle the cold-start problem of dialogue systems. for the company, such a system would greatly improve their service portfolio. we treat this question on an exemplar dataset derived from a german sme. the higher-level goal of this company is to digitalize and automate travel bookings. they collaborate with travel agencies, where they receive travel requests by e-mail and respond with a list of recommendations. these e-mails contain diverse, often unstructured pieces of information in various amounts and levels of detail, increasing the complexity of translation immensely. a dialogue system is well-suited for booking scenarios since it allows for filling the needed slots and offers the possibility to, bidirectionally, ask for additional information, make proposals, and change previous slots. this interactivity mimics the interaction between a user and a respective human counterpart much more closely than an e-mail. moreover, such a system would speed up processes because the response time in synchronous communication channels (chats) would be generally shorter than for asynchronous commonuication (emails). the goal of the intended tod system would be to assess the user's needs and wants. the final, legally binding confirmation of the booking would happen through a second communication channel. training a tod system on e-mails directly is not possible since, e.g., the format and style don't fit, they lack the chatbot speaker role, and they are not annotated. the translation from e-mails to dialogues is complex and infeasible with traditional algorithms for multiple reasons. firstly, one has to be able to identify all domain-specific relevant information within the e-mail, which is a nlu task. then, one has to generate user and system utterances, which entails all the conundrums of nlg. moreover, this nlg will in many cases need to include new, contextually relevant information that was not given in the e-mail. for example, if the e-mail only contains the destination, the chatbot would have to ask for, e.g., the travel period. therefore, some information will need to be invented, i.e., hallucinated. the recent advances in llms could offer an elegant solution to all of these tasks, with guided hallucinations even being desired to some extent. to enable this project, we defined a specific ontology (cf. tab. 4 in the appendix a) will be the basis for the llm prompts and dialogue annotation. to the best of our knowledge, there is no current method that could be used for the problem that we tackle. hotel: bahia del sol desired hotel: bahia del sol email address: example@e-mail.com name: <@xmath0: +49 +49 12312312312 departure airports: dtm, dus, cgn, pad travel period: october 1st, 2021 -october 8th, 2021 travel duration: 1 week participants: 2 adults meals: all inclusive asking price: up to 750 request: a double room all inclusive with a balcony if you have a sea view, preferably a direct flight to turkey or mallorca user: "i would like to book a hotel with all-inclusive." // inform(@xmath1, inform(@xmath2: "sure, that's no problem. do you have any preferences for the hotel?" // act_require_more(hotel) user: "yes, it should have a direct flight to turkey or mallorca." // inform(@xmath3, inform(@xmath4, mallorca]) bot: "okay, do you want to stay in a double room?" // request(hotel_room_type) user: "yes, please." // inform(@xmath5: "do you have any preferences for the room number?" // act_require_more(@xmath6: "no, that's okay." // inform(@xmath7: "i recommend the bahia del sol hotel, which is located directly at the beach in the town of los cristianos. it has 4 stars and is all inclusive. would you like to book it?" // inform(@xmath8, inform(@xmath9, inform(@xmath10, inform(@xmath11, act_booking(@xmath12: "can you please give me the address of the hotel?" // request(hotel_address) bot: "sure, the bahia del sol hotel is located at av. de las américas, s/n, 38650 playa de las américas, las palmas, spain. would you like to book it?" // inform(@xmath13. de las américas, s/n, 38650 playa de las américas, las palmas, spain), act_booking(@xmath14: "yes, please!" // act_booking(@xmath15: "great, can you please provide me with your e-mail address, so i can send you the booking information?" // request(user_e-mail) user: "info@example.com" // inform(user_e-@xmath16.com) bot: "i have sent you all necessary information. is there anything else i can do for you?" // act_information_sent(yes), act_require_more(@xmath17: "no, thank you. that's all." // inform(@xmath18, act_require_more(@xmath19: "you're welcome, bye!" // act_general(bye) 2.1 existing data and pre-processing 2.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 umap dimension 1 2 4 6 8 10 12 14 umap dimension 2 cluster short elaborate figure 3: the clustering of the e-mails used for the train split. we first convert the e-mails with tf-idf and then encode them with umap to build the clusters. it is clear that the short e-mails are the majority. the existing data comes from an uncurated database dump of e-mail requests with highly heterogeneous styles. they range from minimal oneline e-mails, e.g., "namibia individual trip", to elaborate, prose-like free texts and e-mails that give detailed information in an enumeration style. we can roughly cluster the e-mails into either short or elaborate, as is shown with in fig. 3 . since the data is raw, it includes a significant amount of noise. for example, out-of-office notifications, empty e-mails, test messages, and even apparent scam attempts. we apply a rigorous rulebased filtering to exclude noise. this affected roughly 10% of the full dataset. moreover, we anonymized the data to remove personal client in-formation. the e-mails are nearly exclusively in german. however, our preliminary experiments showed that the used llm has poor performance on german text. we therefore applied one further step of pre-processing, translating the e-mails from german to english with the google translate api foot_0 . finally, we construct the train, validation and test datasets by randomly sampling 1500, 150, and 200 e-mails, respectively, ensuring each e-mail is part of only one split. in summary, our data preparation consists of filtering, anonymization, translation and sampling for the data splits. ## background and related work we will now describe related work for tod systems, data augmentation and data style translation. tod systems and datasets. tod systems were traditionally implemented by solving each subtask separately @xcite . with the publication of large datasets, the field has moved towards deep learning-based systems such as @xcite al. (2020); peng et al. (2021a) ; @xcite . benchmark datasets include, e.g., multi-woz @xcite , kvret @xcite and sgd @xcite . data augmentation for dialogues. data augmentation describes the sourcing of synthetic data by applying certain transformations to existing data in order to increase the amount of training data and the model's generalization ability @xcite . approaches include backtranslation @xcite , incorporation of external datasets @xcite , simulating dialogues based on schemata @xcite , graphs @xcite , framing it as a text infilling task @xcite or using specially trained generator models @xcite . nowadays, this line of research has also turned to llms. these methods include, for example, paraphrasing templates, using seed data or adding miscommunications to the dialogues @xcite @xcite @xcite . recently, model collapse (shumailov et al., 2024) has been discussed, where a model's performance degrades with every iteration of it being trained on model-generated data. one way to counteract this is by combining real and synthetic data @xcite , which our method does by utilizing the human-written e-mails. ## method our approach uses instruction-tuned llms to generate dialogues based on monologue e-mails and subsequently annotate them. the llms undergo fine-tuning to solve these tasks. crowd workers provide gold-standard labels for the test dataset. the following sections provide a detailed breakdown of the two phases and finally explain the dataset sourcing. ## evaluation to evaluate monotodia, we evaluate (i) the dialogue generation and annotation in isloation and (ii) the usefulness of the monotodia dialogues for training tod systems. ## results and discussion dialogue generation. the outcomes of the crowd worker evaluation is summarized in tab. 2, showing the average ratings for each question. they show that the generated dialogues have high quality, achieving an average rating of at least 4 out of 5 in nearly all tested criteria, with the lowest rating being 3.98. we see that even after our filtering, the ## conclusion this study investigates the feasibility and efficacy of using llms to translate e-mails into annotated task-oriented dialogues for the travel booking domain. by fine-tuning a state-of-the-art open-source llm, performing extensive human assessment and empirical analysis, we have shown that the generated dialogues are of good quality and suitable for downstream training of tod systems. even for input e-mails that lack all necessary information, the dialogues achieved good scores. note that the published dataset uses train and validation annotations predicted from fine-tuning on 220 goldstandard dialogues (f 1 a ) to provide higher-quality annotations. nevertheless, we observe that even a smaller dataset of only 20 examples can be a sufficient foundation for the training of tod systems. the evaluation results show that the generated dialogues closely resemble real conversations, contain relevant information, and that the bot in the conversations is helpful in achieving the user's goal. furthermore, the llm closely followed the rules to generate the dialogue based on the e-mail. these results are consistent with other studies on synthetic dialogues @xcite @xcite , even though they follow different paradigms and do not translate from existing data. overall, the findings suggest that this approach holds promise for addressing the challenges of data scarcity in training tod systems. even though our study is limited to only e-mails, we think that by leveraging existing data sources, such as e-mails, it support tickets, or transcribed calls, and employing modern llms, companies can thus overcome barriers to deploying tod systems in their service portfolios. moreover, we had to translate the e-mails and proceed with english dialogues, which will need to be translated back into german for the use case in the cooperating company. llms that perform better on german are thus of high interest. we publish the resulting dataset to support future research.