LUCID : LLM -Generated Utterances for Complex and Interesting Dialogues (2024)

Abstract: Spurred by recent advances in Large Language Models (LLMs), virtual assistants are poised to take a leap forward in terms of their dialogue capabilities. Yet a major bottleneck to achieving genuinely transformative task-oriented dialogue capabilities remains the scarcity of high quality data. Existing datasets, while impressive in scale, have limited domain coverage and contain few genuinely challenging conversational phenomena; those which are present are typically unlabelled, making it difficult to assess the strengths and weaknesses of models without time-consuming and costly human evaluation. Moreover, creating high quality dialogue data has until now required considerable human input, limiting both the scale of these datasets and the ability to rapidly bootstrap data for a new target domain. We aim to overcome these issues with LUCID, a modularised and highly automated LLM-driven data generation system that produces realistic, diverse and challenging dialogues. We use LUCID to generate a seed dataset of 4,277 conversations across 100 intents to demonstrate its capabilities, with a human review finding consistently high quality labels in the generated data.

## introduction as ai virtual assistants become more sophisticated, there is an increasing need for dialogue datasets with more challenging conversational phenomena for both fine-tuning and evaluation. existing datasets include multi-turn, multi-intent and multi-domain conversations @xcite , in addition to multilingual datasets @xcite @xcite . however, in each case, the number of intents covered is relatively small. moreover, the conversational phenomena included in these datasets are often limited in * work undertaken while author was an intern at apple 1 please see http://github.com/apple/ml-lucid-datagen for the data and project code conversation extract: lucid user sure, i'd be happy to help with that. can you please tell me the name of the film you'd like to review? i want to review a film it seems like you're busy at the moment. let's continue this conversation after dinner. bobby, stop playing with your food and eat your dinner! lucid lucid yes, that's correct. my review is: i found the storyline of this movie a bit hard to follow. ## method lucid decomposes the data generation process into 14 individual llm calls, described here as stages, creating manageable steps that llms can perform accurately. alongside our if in doubt, discard validation, reducing the complexity of each llm call helps to ensure the quality of our generated data. the data generation process consists of four main components (see figure 2 ): the generation of intents (stages 1-2), a conversational planner (stages 3-8), turn-by-turn generation of conversations (stages 9-12), and our validation process (stages 13 and 14). the turn-by-turn data generation involves a user llm agent interacting with a system llm agent, which in turn interacts with a mock back-end created for each intent. ## baseline results we train t5 @xcite and flan-t5 (chung et al., 2022) baseline models on lu-cid, evaluating on both our in-distribution and out-of-distribution test sets. when retrieving intent schemas, a sentence-bert @xcite model is used to encode the tool name and the last user utterance, choosing the tool with the highest cosine similarity. as expected, the joint goal accuracy is considerably higher when evaluating on the seen test set compared to the unseen test set, with accuracy scores of 67.5% and 42.2% respectively for a t5-base model (see ## conclusion we introduce lucid, a pipeline of llm calls which is designed to create high quality and linguistically sophisticated dialogue data. lucid involves an extensive validation process, including three validator llms that discard conversations where there is any disagreement. to demonstrate the quality of the data produced, we generate a seed dataset of 4,277 dialogues, consisting of 92,699 turns, with a wide variety of challenging conversational phenomena. the generated system labels in lucid prove to be highly accurate, with only 1% of conversations containing a labelling error. we make our code available to facilitate larger scale, high quality data generation.