Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction (2025)

Abstract: The improvement of LLMs’ instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm—Web as Instruction and Web as Response—where each web document is designated as either the input or output role to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort.

## introduction large language models (llms) @xcite @xcite have become integral across a myriad of applications, demonstrating exceptional performance on diverse tasks by effectively following instructions @xcite . their remarkable performance largely stems from supervised finetuning (sft) @xcite on instruction-response pairs. this process empowers llms to produce customized outputs when pro-* work done during the internship at huawei noah's ark lab. data and code: https://github.com/yjiangcm/webr . vided with specific instructions, facilitating their adaptation to novel tasks without prior exposure. a fundamental challenge in advancing the instruction-following capabilities of llms lies in the collection of high-quality instruction-tuning (it) data. early approaches primarily rely on human experts to manually generate and curate it data @xcite , which is both time-intensive and resource-heavy. to mitigate these limitations, semi-automated synthetic methods @xcite @xcite leverage llms to expand small, human-annotated seed datasets using fewshot prompting techniques. while effective, the performance of these methods is highly sensitive to prompt engineering and the careful selection of seed examples @xcite . more recently, fully automated synthetic methods, such as we-binstruct @xcite and instruction backtranslation @xcite , have emerged as scalable alternatives that eliminate human involvement by synthesizing it data based on web-scraped doc-uments. these methods, however, often operate under strong assumptions about the structure and content of raw web data, such as the availability of explicit question-answer pairs or minimal irrelevant content. consequently, they can only handle a limited scope of web documents, restricting their diversity and leading to suboptimal performance across various tasks. to overcome these limitations, we propose web reconstruction (webr)-a novel framework that synthesizes high-quality it data from raw web documents with minimal assumptions on web and no reliance on human annotations, enabling broader adaptability and improved performance. unlike backtranslation @xcite , which directly treats web content as a response, or webinstruct @xcite , which extracts qa pairs, webr introduces a novel paradigm by conceptualizing web reconstruction as an instructiontuning data synthesis task. at its core, webr aims to transform raw, noisy web documents into human-preferred, response-like outputs through a dual-perspective paradigm. each web document is designated as either an instruction or a response, triggering the reconstruction process: (1) web as instruction introduces a first-of-its-kind web rewriting approach in it data synthesis, where raw web document is concatenated with a synthesized rewrite request to serve as a complete instruction; we apply webr to the llama3-70b-instruct and gpt-4o-mini models, creating two 100k-sample it datasets: webr-basic and webr-pro. to validate their effectiveness, we train various llms, including llama3-8b-base and qwen2.5-1.5/3/7/14bbase, and evaluate them on over ten widely used benchmarks. our experiments provide key contributions and insights into it data synthesis: ## related work the synthesis of high-quality instruction-tuning (it) data @xcite @xcite can be broadly classified into three main categories: human-crafted method primarily involves employing professionals to create instructions, as seen in datasets like super-ni @xcite and dolly @xcite . while these datasets offer high-quality content, their size is constrained by the significant costs associated with manual creation. alternatively, approaches like sharegpt @xcite and wild-chat @xcite leverage user interaction logs with llms to collect human-generated instructions. however, this method risks incorporating toxic or undesirable content @xcite . semi-automated synthetic method uses llms to generate synthetic it datasets by starting with a small set of human-annotated seed data and expanding them through few-shot prompting. notable methods include self-instruct @xcite , alpaca @xcite , and evol-instruct @xcite . while these techniques enable largescale data generation, the diversity of the synthesized data is often constrained by the quality and variety of seed examples @xcite . ## web reconstruction prior fully automated synthetic methods often rely on strong assumptions about the structure and content of raw web documents-such as the presence of explicit question-answer pairs, minimal irrelevant content, or appropriate expressions-necessitating complex preprocessing steps like retrieval and filtering. in contrast, we introduce the web reconstruction (webr) framework, which leverages a powerful, off-the-shelf llm to overcome these limitations by directly reconstructing unstructured and noisy web content into highquality, response-like outputs. as shown in figure 2 , webr comprises two core strategies: (1) web as instruction, where raw web content is concatenated with a synthesized rewrite request to serve as a complete instruction, guiding the generation of a reorganized, coherent response; (2) web as response, where a latent instruction is inferred by treating raw web content as a response, enabling reconstruction through the llm's initial rollout and subsequent refinement. by adopting this dual-branch approach, webr efficiently generates high-quality instructionresponse pairs, ensuring contextually appropriate outputs while eliminating the need for extensive preprocessing. ## analysis 6.1 dataset analysis of webr diversity. we utilize a quantitative measure of diversity: (1) we randomly sample @xmath0, 000 instructions from each dataset and encode them using the all-mpnet-base-v2 foot_0 embedding model; (2) we compute the average cosine similarity between all embedding pairs and define embedding diversity as 1 -1 c(n,2) ∀@xmath1, e j ), where higher values indicate greater diversity. ## conclusion in this paper, we present web reconstruction (webr), a fully automated framework for synthesizing high-quality instruction-tuning (it) datasets. harnessing the richness of raw web content, we conceptualize web reconstruction as an instructiontuning data synthesis task via a novel dualperspective paradigm-web as instruction and web as response-where each web document is des-ignated as either an instruction or a response to trigger the reconstruction process. extensive experiments show that webr-generated datasets consistently outperform state-of-the-art baselines across four instruction-following benchmarks and six diverse downstream tasks. furthermore, webr exhibits exceptional compatibility, data efficiency, and scalability with existing datasets, underscoring its potential as a versatile tool for advancing instruction-driven llm training.