Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering (2024)

Abstract: Few-shot learning for open domain multi-hop question answering typically relies on the in-context learning capability of large language models (LLMs). While powerful, these LLMs usually contain tens or hundreds of billions of parameters, making them rather inefficient at inference time. To improve performance of smaller language models, we propose a data synthesis framework for multi-hop question answering that requires less than 10 human-annotated question answer pairs. Our framework depends only on rich, naturally-occurring relationships among documents and is built upon the data generation functions parameterized by LLMs and prompts. We synthesize millions of multi-hop questions and claims to finetune language models, evaluated on popular benchmarks for multi-hop question answering and fact verification. Empirically, our approach improves model performance significantly, allowing the finetuned models to be competitive with GPT-3.5 based approaches while being almost one-third the size in parameter count.

## introduction few-shot learning for open domain multi-hop question answering seeks to answer complex questions by iteratively retrieving relevant information with a handful of human-annotated question answer pairs. it has become increasingly popular for evaluating the abilities of grounding to factual and up-to-date information @xcite and the reasoning capabilities @xcite of large language models (llms). recent approaches in this area typically rely on in-context learning @xcite where llms are prompted to retrieve relevant information using external search tools @xcite . while powerful, the in-context learning capability usually emerges when llms have billions of parameters and improves as llms become larger in size @xcite . this property makes llms expensive to experiment with even for inference. in this work, we propose a data synthesis framework for multi-hop question answering (mqa) that allows for improving smaller language models with less than 10 human-annotated qa pairs (see @xcite 1 for an overall pipeline of our approach). the framework seeks to generate mqa data using documents that are related in different aspects, e.g., sharing similar topics, providing extra information about entities, or talking about events occurred in sequence. this framework is general in that (1) the relationships among documents are naturallyoccurring, covering a diverse set of reasoning types; and (2) the data generation pipeline depends on few hand-crafted, task-dependent features. specifically, we choose to use wikipedia as our data sources due to its comprehensive coverage of knowledge and use hyperlinks to capture rich document relationships beyond topic similarity. we start from document pairs that are either topically similar or connected by hyperlinks, then we prompt llms to perform three generation tasks: question generation, question answering, and query generation. we do so by simply changing the format of prompts while re-using the same set of qa pairs. finally, we verify the quality of queries against retrieval corpora using a neural retriever. we also show that this framework can be easily adapted to other tasks, e.g., fact verification, as demonstrated in our experiments. unlike prior work on data synthesis for mqa @xcite , which often depends on carefully designed templates to facilitate complex question generation, limiting the diversity of types of reasoning in their generation questions, our approach requires minimal hand-crafted features as it is built upon llms through prompting. in contrast to most work on data synthesis with llms @xcite , inter alia) that primarily uses a single data generation function per task, our data generation process involves multiple figure 1 : an illustration of the overall pipeline of our proposed approach. each data instance in our synthesized dataset consists of a question, queries and their corresponding retrieved documents, and an answer. we first prompt llms to synthesize questions and queries, finetune models on the synthesized data, and then evaluate the finetuned models on downstream tasks that require iteratively querying retrieval corpora. in experiments, we use a frozen llama 65b @xcite to synthesize approximately 1.5 million multi-hop questions and 1.9 million claims, each of which comes with with queries and answers. to validate the effectiveness of the synthetic data, we finetune 7b-and 65b-parameter llama models on it and then evaluate the finetuned models on three popular multi-hop question answering benchmarks and one fact verification dataset. empirically, we observe that finetuning on the synthetic data drastically improves model performance, allowing our finetuned llama 7b to achieve better performance than vanilla llama 65b. crucially, since the data is synthesized by llama 65b, the improvement from llama 65b essentially comes from the effect similar to selftraining. when comparing to prior work on question and query generation, we show that our approach achieve better performance while requiring less hand-crafted features. analysis reveals that finetuning on the synthetic data helps models of different sizes, particularly showcasing greater benefits for smaller models. moreover, we find that automatic filtering steps and having diverse relationships among documents are crucial in improving model performance. to summarize, our contributions are: ## approach we seek to synthesize training data for multi-hop question answering using a handful of human annotations. our data synthesis pipeline leverages naturally-occurring relationships among documents and the powerful reasoning abilities of llms. each generated data instance contains a question, up to two queries, and an answer. we then finetune models on the generated data. the data generation process consists of four main steps: question generation, question answering, query generation, and query verification. to achieve this, we use a frozen llama 65b and parameterize the underlying data generation functions with different prompts. 1 as shown in figure 1 right, our approach can be broken into following steps: we note that this entire process uses the same set of examples, consisting of up to 10 humanannotated data instances. we use these examples to create prompts for the tasks specified in steps 2, 3, and 4. we describe each step in detail below. ## conclusion we propose a llms-based data synthesis framework for open domain multi-hop question answering that demands less than 10 qa pairs. the framework requires less hand-crafted features than prior work while still achieving better performance. we show that our approach is general by extending to fact verification tasks. our results on three multihop question answering and one fact verification benchmarks show that our approach leads to signif-icantly smaller models that rival the performance of previous methods. the analysis shows (1) the importance of the filtering steps and diverse relationships among documents; and (2) our approach benefits models of various sizes.