M eta S ynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation (2025)

Abstract: Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data generated using larger Language models. Questions remain about leveraging synthetic data for other use cases, such as adapting LLMs to specific domains. A key limitation of synthetic data is low diversity , which negatively impacts its downstream applicability for improving other models. To address this, we propose MetaSynth, a method for generating synthetic data that enhances diversity through meta-prompting, where a language model orchestrates multiple “expert” LLM agents to collaboratively generate data. Using only 25 million tokens of synthetic data generated with MetaSynth, we successfully adapt a well-trained LLM (Mistral-7B) to two specialized domains–Finance and Biomedicine–without compromising the capabilities of the resulting model in general tasks. In addition, we evaluate the diversity of our synthetic data using seven automated metrics, and find that it approaches the diversity of LLM pre-training corpora.Continually pre-training Mistral-7B with MetaSynth notably outperforms the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in Biomedicine. The same model shows degraded performance when trained on data generated using a template-based prompt, even when the template includes prior generations and varying In-Context exemplars of real data. Our findings suggest that a few million tokens of diverse synthetic data without mixing any real data, is sufficient for effective domain adaptation when using MetaSynth.

## introduction human generated public text data cannot sustain the continued scaling and expansion of large language models (llms). it has been argued by @xcite that the available stock of public human text data will be fully utilized by 2028 if current llm development trends continue, or earlier if llms are trained on more data than is compute optimal. this is evidenced in e.g., llama 3 @xcite which uses one order of magnitude more data compared to only two year old estimates of compute optimal large language models @xcite . as a potential remedy, synthetic data generated with llms has shown remarkable potential to alleviate the impending issue of data scarcity for future model scaling. however, low diversity is a key issue in any type of synthetic data. in this work, we hypothesize that there are two prominent reasons that affect the diversity of data synthesized by llms: a) the choice of seed instances used to initialize data generation and b) the prompts used, which commonly follow predefined templates, where variation in the prompt is mainly introduced via placeholders whose content is populated dynamically. examples of data generation methods which use template-like prompts with variation include: self-prompting @xcite , @xcite , @xcite and explore-instruct @xcite , among others. we contend that this variation yields limited diversity. for instance, generating a collection of domain-specific (e.g., financial) texts with similar prompts results in repetitive sentence structures-many texts begin with lexical patterns such as "in today's ever-changing financial landscape" or "as the financial world evolves"-and often contain recurring phrases, and generic buzzwords. recently, @xcite find that metaprompting @xcite approaches -where an llm itself writes the prompts to solve a problem -can elicit more diverse and creative outputs, significantly improving problem-solving capabilities for mathematical and algorithmic reasoning tasks, largely due to the feedback, self-verification, chain-of-thought @xcite , and planning dynamics inherent in these approaches. it has also been shown that using an optimized meta-prompt can improve the quality and downstream effectiveness of llm generated synthetic data @xcite . we argue that a key use case for synthetic data arises when abundant real data exists in the form of pre-training corpora, but one wishes to effectively tailor an llm to a specific domain using only a small amount of carefully generated synthetic data @xcite . in this work, we investigate data-efficient domain adaptation through meta-prompting, where a language model is instructed to act as a supervisor that writes specialized prompts for other models to collaboratively generate diverse data. our contributions are as follows: ## measuring the diversity of generated synthetic data the premise of this work is that diverse data is high quality data. thus, in lieu of human judgment of diversity; it is necessary to use an appropriate set of automated metrics which can quantify the diversity of llm generated data such that these measures also align with human notions of variability and diversity. ## analysis to evaluate the utility of our instruction synthesizer (metasynth-instruct) in creating instructions for more general tasks, we conduct the following analyses: creating data for fine-tuning encoders we adapt our instruction synthesizer to generate synthetic data that emulates datasets used in encoder lm evaluation. we modify the task description in the meta-prompt (appendix l.4) to instruct the meta-lm to generate synthetic training instances resembling each of three finance datasets-headline news (sarcasm detection), fiqa-sa (aspect-based sentiment analysis), and financial phrasebank (sentiment analysis)-selected for their simplicity and prior use in @xcite al. (2023)'s work. for each dataset, we generate a small set of synthetic instances (refer to ## related work previous work on generating synthetic data with llms has primarily focused on post-training data synthesis, particularly for conversational data or instructions @xcite @xcite @xcite inter alia. recent work like agentinstruct (mitra et al., 2024) uses predefined taxonomies and agentic workflows to generate diverse instruction-response pairs from real corpora. while similar to our approach in using iterative refinement, our method differs by leveraging the meta-model's reasoning to dynamically select synthesis flows rather than sampling from fixed taxonomies. unlike post-training approaches that compute loss only on responses, our method aligns with pre-training approaches which compute the loss on both prompts and responses. we focus on increasing model domain knowledge by targeting continual pre-training (cpt) over supervised finetuning (sft) based on observations that knowledge is primarily learned during pre-training, while sft only improves in-distribution task performance and can cause forgetting of unused abilities and domain knowledge (zhou et al., 2023; sun and dredze, 2025; ke et al., 2025). other agentic approaches for synthetic data generation include unigen (wu et al., 2024), which generates sft data in the format of a target dataset (unlike our method which does not require a target dataset), matrix (tang et al., 2025), which focuses on synthesizing preference-tuning data for instruction-following, not pre-training, and person-ahub (ge et al., 2024), which samples 1b personas from 10 14 tokens of web text and then uses a template prompt e.g., "create data with persona" to synthesize instances (given the large scale of real data used in personahub, it is not comparable to our method). we note that non-agentic prompting techniques for extracting diverse data from llms such as hayati et al. (2024) and wong et al. (2024), and human-in-the-loop methods such as chung et al. (2023) are complementary to our approach. in this work we demonstrate that it is possible to generate useful synthetic data while using real data resource-efficiently: we utilize ∼26.5m tokens of real data from domain-specific common crawl splits (approximately 50k documents) to synthesize 25m tokens of synthetic data (∼47k documents). this is substantially lower than prior work such as adaptllm @xcite which used billions of tokens of real corpora (5.4b medical, 1.2b finance) for synthetic data generation. additionally, our method is unsupervised, generating instructions from synthetic texts, whereas instruction-pretraining @xcite ) leverages an instruction synthesizer trained on at least 1b tokens of real corpora. ## conclusion we propose metasynth, a method that leverages meta-prompting and agentic scaffolding to generate measurably diverse documents and instructions. we demonstrate its efficacy by synthesizing diverse data and then continually pre-training mistral-7b on it, yielding significant improvements in two domains, without degrading the model on general tasks.