MANTA : A Scalable Pipeline for Transmuting Massive Web Corpora into Instruction Datasets (2025)

Abstract: We introduce MANTA, an automated pipeline that generates high-quality large-scale instruction fine-tuning datasets from massive web corpora while preserving their diversity and scalability. By extracting structured syllabi from web documents and leveraging high-performance LLMs, our approach enables highly effective query-response generation with minimal human intervention. Extensive experiments on 8B-scale LLMs demonstrate that fine-tuning on the MANTA-1M dataset significantly outperforms other massive dataset generation methodologies, particularly in knowledge-intensive tasks such as MMLU and MMLU-Pro, while also delivering superior performance across a broad spectrum of tasks. Moreover, MANTA supports seamless scalability by allowing the continuous integration of web corpus data, enabling expansion into domains requiring intensive knowledge.

## introduction the scalability of the dataset size is a critical factor not only in the pre-training stage but also in the instruction fine-tuning stage @xcite @xcite @xcite . a major challenge in large-scale instruction fine-tuning dataset generation is that directly constructing a supervised dataset, consisting of user queries and assistant responses, is highly human-labor intensive @xcite @xcite . to more effectively construct high-quality largescale fine-tuning datasets, recent researches have been conducted on developing automated pipelines with less human intervention @xcite @xcite @xcite . in this context, we aim to explore how the vast and diverse knowledge within massive web corpora can be effectively synthesized into an instruction dataset. in this paper, we present a dataset generation pipeline for instruction fine-tuning from massive web corpora. our proposed pipeline, manta, preserves the diversity of web corpora and scales up to the size of the source web corpora, enabling the construction of large-scale high quality instruction fine-tuning datasets with minimal human effort. this is achieved through the systematic employment of publicly accessible high-performance llms and web corpora filtered for educational value. manta pipeline starts by encapsulating the core knowledge of each large-scale web document into a json-formatted structured syllabus, as shown in figure 1 . this approach enables us to achieve both the high quality derived from edu-cational value and the robustness in generating instructions facilitated by the knowledge-friendly syllabus format. the large-scale syllabi, constructed with near-zero human effort, offer a significant advantage over previous massive instruction dataset generation methodologies in that their topic distribution is not confined to predefined taxonomies @xcite , curated seed examples @xcite , or the llm's learned knowledge base @xcite . by utilizing a publicly accessible fine-tuned llm, manta pipeline generates a large-scale supervised dataset composed of query-response pairs generated from the syllabi. additionally, we propose a method for generating more complex instruction pairs. syllabus fusion allows us to increase instruction difficulty by combining multiple syllabi, resulting in enhanced performance of models trained on this dataset. multi-turn expansion enables the generation of realistic follow-up conversations, improving the usability of the trained model. we rigorously validate the manta pipeline through extensive experiments by fine-tuning various pre-trained llms with 8b parameters on 1 million manta-generated data points, while also conducting comparative experiments with 1 million data points from other massive dataset generation methodologies. in our experiments, three pretrained models-mistral-3-7b-v0.3 (jiang et al., 2023), llama-3.1-8b @xcite , and exaone-3.5-7.8b (lg ai research, 2024)-were used for baseline. this results in a significant performance gap compared to fine-tuning on other massively generated datasets, demonstrating a clear performance advantage. a notable finding is that the performance gap is particularly pronounced in mmlu @xcite and mmlu-pro @xcite , which require diverse knowledge. the models fine-tuned on the manta-1m not only excel in the academic knowledge domain but also achieve high performance across various domains, spanning from a mathematics such as @xcite to instructionfollowing ability like mt-bench @xcite . in addition, the diversity and difficulty analyses conducted on the manta-1m dataset itself suggest that the manta pipeline effectively mimics the topic distribution of massive web corpora while generating more challenging queries. we highlight our main contributions as follows: ## related work as the importance of diverse and large-scale instruction tuning datasets becomes more pronounced, recent studies predominantly focus on synthesizing mass datasets handling diverse domains by prompting llms. starting from a few data points used as the seed dataset, many studies utilize language models' in-context learning ability to generate new datasets @xcite @xcite . while these methods are scalable, they need well-crafted human-generated seed demonstrations. another line of research synthesizes instruction dataset based on the pre-defined topics or taxonomies of the instruction. ultrachat @xcite first builds the list of topics or materials used for starting points for building instruction dataset by interacting with llms. glan @xcite also generates datasets by building hierarchical taxonomy of human knowledge. orca3 @xcite converts raw text according to 17 defined skills to used as seed skills to generate dataset. while these methodolgies have advantage to focus on pre-defined abilities that the dataset builders expect their llms to have, they are also required to enumerate the list of topics. furthermore, the list of the capabilities or topics is limited to the knowledge of a few human experts, even they utilize powerful llms when brainstorming. to bypass construction of seed datasets or knowledge taxonomy, recent studies leverage enormous amount of web corpus to extract and synthesize instruction dataset. @xcite collects raw document from web corpus and directly extracts question-answer pairs from them. while they rely on web corpus to represent world knowledge distribution, the web corpus that is able to be used for source of the instruct is limited as the format or quality. on the other hand, since our method compresses the raw document and extracts syllabus from it, manta pipeline is not constrained to the specific format or characteristics of the document. ## manta pipeline we develop a massive instruction data generation pipeline, manta, that consists of three major subprocesses. first, we reconstruct wellformatted texts playing role as syllabus that encapsulate the core knowledge from crawled web documents. then we generate query-response pairs from the syllabus. lastly, we enrich the generalization performance through difficulty enhancement and multi-turn expansion. manta prioritizes generating a high-quality instruction fine-tuning dataset from large-scale web corpora with minimal human effort. by utilizing this pipeline, we have generated 200k of syllabus and 3m query-response pairs from them. the overall pipeline for these processes is shown in figure 2 . ## conclusion we present manta, an automated pipeline designed to generate high-quality large-scale instruction fine-tuning datasets from massive web corpora, while maintaining their diversity and scalability. with a syllabus-formatted structure distilled from massive web corpora, the manta pipeline robustly generates effective query-response pairs with minimal human effort. extensive evaluations on 8b-scale llms show that fine-tuning on the manta-1m dataset consistently outperforms other large-scale dataset generation approaches, excelling in knowledge-intensive tasks like mmlu and mmlu-pro, while also demonstrating strong results across a wide range of benchmarks.