Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts (2024)

Abstract: Automating data generation with Large Language Models (LLMs) has become increasingly popular. In this work, we investigate the feasibility and effectiveness of LLM-based data generation in the challenging setting of source-grounded information-seeking dialogs, with response attribution, over long documents. Our source texts consist of long and noisy meeting transcripts, adding to the task complexity. Since automating attribution remains difficult, we propose a semi-automatic approach: dialog queries and responses are generated with LLMs, followed by human verification and identification of attribution spans. Using this approach, we created MISeD – Meeting Information Seeking Dialogs dataset – a dataset of information-seeking dialogs focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance compared to off-the-shelf models, even those of larger size. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort.

## introduction source-grounded information-seeking dialogs allow users to efficiently navigate within a given knowledge source and extract information of interest. in this conversational setting, a user interacts with an agent over multiple rounds of queries and responses regarding the source text @xcite @xcite . to train effective agent models, quality dialog datasets are essential. the prominent (manual) technique for creating dialog datasets is the wizard-of-oz (woz) method @xcite , in which * these authors contributed equally. figure 1 : iterative dialog generation flow. in each turn, a query prompt guides the llm to generate a user query given the transcript, the accumulated dialog history, and a query template. then, a response prompt, accompanied by the full context so far, generates the agent response. iterating this automatic process yields a full dialog, which is then validated by annotators, who further augment it with response attributions. previous work has explored the application of llm-based dialog generation in domains like everyday conversations @xcite or taskoriented dialogs @xcite . in this work, we investigate potential automation of the woz process to create sourcegrounded information-seeking dialogs. while llms can generate dialog content, attribution generation currently requires human involvement for reasonable quality. therefore we propose a semi-automatic approach: using prompts to guide llm generation of queries and responses, followed by manual attribution and validation of response correctness (figure 1 ). by comparing our semiautomatic approach to the traditional, fully-manual woz method, we demonstrate that a model trained on our data achieves comparable response generation quality while improving attribution and reducing costs. we apply our methodology to create misedthe first dataset for information-seeking dialogs over meeting transcripts, supporting the use-case of users catching up on meetings they have missed. meeting transcripts present additional challenges as they often contain disfluencies, interruptions, and off-topic comments-challenges that are also representative of a broad range of real-world conversational settings. existing datasets in the meeting domain provide summarization and questionanswering data (e.g., @xcite , but none supports multi-turn dialogs over meeting content. the dialog setup introduces additional complexity, as each query and response must consider the shared dialog state (e.g., previously shared information) in addition to the transcript, to maintain a coherent dialog flow. we further present an evaluation framework and few baseline models, whose evaluation demonstrates the benefit of training with our mised data. in summary, our main contributions are: (1) we explore the feasibility of using llms for data generation in source-grounded information-seeking dialogs, demonstrating improved overall quality and efficiency compared to a fully manual approach; ## related work this section provides background on the two research areas which our work bridges: source-grounded information-seeking dialogs ( §2.1) and summarization and question answering (q&a) over meeting transcripts ( §2.2). ## source-grounded information-seeking dialogs: task definition as described above, we focus on generating data to train agent models for the task of source-grounded information-seeking dialogs. the agent model task (demonstrated in figure 2 ) is defined as follows: at each dialog turn the user issues a query about the source text. the agent model receives the source text, the preceding dialog history, and the current query, and is tasked with generating a response and providing the supporting attributions. each attribution is a consecutive span in the source text. ## dataset creation methodology this section details our semi-automatic dataset creation methodology, as applied to meeting transcripts (figure 1 ). we first automatically generate dialogs using a pre-trained llm ( §4.1), simulating the typical woz process. then we employ human annotators to generate response attributions, a task that currently necessitates human expertise. as part of the attribution process, annotators also verify the accuracy and validity of the generated queries and responses ( §4.2). ## datasets this section describes the mised dataset ( §5.1), constructed using our semi-authomatic method ( §4). we also present a smaller independent dataset created using the fully manual wizard-of-oz methodology ( §5.2), created to assess whether mised data is comparable to human-generated data. finally, we present a manual assessment of mised response quality ( §5.3). ## evaluation methodologies we evaluate the agent models along two dimensions: the quality of the generated responses ( §6.1), and the accuracy of the provided attributions ( §6.2), through both automatic and human evaluations. ## conclusion in this paper, we investigate llm automation to generate source-grounded information-seeking dialog datasets. we introduce a method to partially automate the woz process using targeted user and agent prompts, followed by human attribution, verification, and potential editing. we apply this method to create mised, the first dataset for information-seeking dialogs over meetings. baseline models and experiments demonstrate the value of mised: finetuninig with mised data brings the same improvement as a comparable amount of the more expensive fully-manual woz data. mised enables the creation of modestly sized finetuned encoder-decoder models with performance approaching much larger pre-trained llms, and further improves the performance of such llms through finetuning. we suggest that our work yields valuable insights about the feasibility of llm-based data generation in the important and challenging case of source-grounded informationseeking dialogs. long texts depend on llm context length our method's reliance on llm prompting poses challenges for long meeting transcripts that exceed the maximal context length of the model. the same problem occurs in the response generation agent task. possible solutions include using llms with larger context lengths (which may be computationally expensive) or integrating a retrieval system as a first stage in the response generation, to condense the transcript and provide the most relevant information to the llm. manually crafted prompt templates our method currently depends on manually crafted prompt templates to represent a broad spectrum of potential queries in the given domain. research into more flexible and diverse query generation strategies could reduce the effort of applying our method in new domains while increasing the generality of the obtained data.