CSLM : A Framework for Question Answering Dataset Generation through Collaborative Small Language Models (2024)

Abstract: Collecting high-quality question-answer (QA) pairs is vital for the training of large language models (LLMs), yet this process is traditionally laborious and time-intensive. With the rapid evolution of LLMs, the potential for leveraging these models to autonomously generate QA pairs has become apparent, particularly through the use of large-scale models like GPT-4. However, the computational demands and associated costs often render such approaches prohibitive for the average researcher. Addressing this gap, we introduce the Collaborative Small Language Model Framework (CSLM), an innovative solution that combines a group of small-scaled, open-source LLMs to collaboratively produce QA pairs. Experiments on datasets of various domains show that CSLM unleashes the full potential of diverse small models to generate high-quality QA pairs, making it accessible to a broader range of researchers.

## introduction the generation of high-quality question-answering (qa) pairs is crucial for enhancing the capabilities of language models across various applications. despite the availability of general domain qa datasets, a significant gap exists in the availability of domain-specific datasets, such as law, medicine, and finance, etc. manual annotation of such datasets is not only laborious and timeconsuming but also entails substantial costs due to the specialized expertise required @xcite . moreover, manual datasets such as squad @xcite often suffer from a lack of diversity, with answers being directly extracted from the source text without the nuance of deeper understanding or context, which restricts the potential of downstream models. to address this, recent research has explored the use of large language models (llms) to synthesize qa pairs from documents or raw corpora @xcite @xcite . however, to generate high-quality qa pairs, largescale models like llama-70b @xcite or closed-source models like @xcite are needed, while reliance on such models is not always feasible due to the substantial computational resource requirements. furthermore, using external apis, like gpt-4, to generate qa pairs introduces privacy and confidentiality concerns, especially when dealing with sensitive data in fields that demand stringent data protection measures. in response to these challenges, we introduce the collaborative small language model framework (cslm) for generating qa pairs. by leveraging a group of smaller, open-source language models connected by a minimal number of extra trainable parameters, cslm harnesses the unique strengths of each model to generate qa pairs that closely match the performance of larger models but with significantly reduced computational requirements. furthermore, cslm allows researchers to maintain control over their data within secure, internal environments, ensuring that sensitive information is protected. we demonstrate the effectiveness of cslm through extensive experiments on various domainspecific texts, showcasing its ability to generate accurate and diverse qa pairs that are not only of high quality but also respectful of privacy and confidentiality constraints, thereby lowering the barrier for organizations and researchers to generate their own datasets. ## conclusion this paper introduces the collaborative small language model framework (cslm), an innovative approach that uses multiple smaller, open-source language models to achieve a performance comparable to larger models, yet with less computational