A uto C onv: Automatically Generating Information-seeking Conversations with Large Language Models (2023)

Abstract: Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years. However, the research is still stymied by the scarcity of training data. To alleviate this problem, we propose AutoConv for synthetic conversation generation, which takes advantage of the few-shot learning ability and generation capacity of large language models (LLM). Specifically, we formulate the conversation generation problem as a language modeling task, then finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process and use it for generating synthetic conversations with high quality. Experimental results on two frequently-used datasets verify that AutoConv has substantial improvements over strong baselines and alleviates the dependence on human annotation. In addition, we also provide several analysis studies to promote future research.

## introduction in information-seeking conversations, users repeatedly ask questions based on their interests, and the dialogue system provides answers to fulfill their information needs @xcite @xcite . this scenario is important for addressing real-world open-ended questions, which requires discussions to explore in depth @xcite , e.g., how to learn more efficiently? though great progress has been achieved in recent years, most existing researches depend on abundant human annotation, which can be highly costly and limited in knowledge coverage. a promising way to alleviate this problem is data augmentation @xcite . traditional methods, including token-level manipulation @xcite method dg data needs eda @xcite ✗ -back-translation @xcite ✗ -seemseek @xcite ✔ large dialog inpainting @xcite ✔ large autoconv (ours) and sentence-level paraphrasing @xcite , improve the linguistic diversity of training data. however, they cannot create conversations grounded on new documents, which are indispensable for dealing with out-of-domain scenarios. another line of research focuses on simulation-based methods @xcite . specifically, they can iteratively generate conversations grounded on new documents based on a span extractor and an utterance generator. nevertheless, both the training of the extractor and the generator still require abundant human dialogues. besides the above ways, @xcite propose dialog inpainting, which creates information-seeking dialogues by inserting utterances between neighboring sentences in documents. one potential risk is the gap between the structure of documents and that of conversations. documents are tighter, while realworld conversations are more open-ended. to alleviate the above issues, we propose a simple yet effective method autoconv for automatically generating information-seeking conversations, which takes advantage of the fewshot learning ability and generation capacity of large language models (llm) @xcite . specifically, we formulate conversation generation as a language modeling task and utilize an llm for generating synthetic conversations grounded on external documents. surprisingly, finetuning with a few human dialogues can help llm capture the characteristics of the information-seeking process (e.g., grounding, question answering) and generate high-quality synthetic conversations. then, we can train a small task model with these dialogues. the differences between autoconv and others are shown in we conduct comprehensive experiments on two frequently-used datasets quac @xcite and coqa @xcite in the low-resource setting, where only dozens of human dialogues are available. the results show that autoconv has substantial improvements over several strong baselines. when scaling up the synthetic dialogues, autoconv has the improvement of up to 5.06 f1 gain compared with directly finetuning, and thus largely reduces the labor force for annotation. in addition, we find that the small task model trained with synthetic dialogues can even surpass finetuned llm with only 1.7% parameters. moreover, we also investigate the impact of decoding strategy and scaling laws for autoconv. ## experiments we conduct experiments on quac @xcite and coqa @xcite , more details about them are shown in appendix a. ## conclusion in this paper, we propose a simple yet effective method, autoconv, which formulates the conversation generation problem as a language modeling task. then, based on a large language model and a few human dialogues, autoconv can generate synthetic dialogues with high quality. experimental results on both quac and coqa verify the effectiveness of autoconv, which alleviates the human efforts for annotation largely. furthermore, we also provide case study and error analysis to prompt future research.