Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation (2025)

Abstract: Large language models (LLMs) with instruction following capabilities have demonstrated impressive problem-solving abilities. While synthesizing instructional data from unsupervised text has become a common approach for training such models, conventional methods rely heavily on human effort for data annotation. Although existing automated synthesis paradigms have alleviated this constraint, they still exhibit significant limitations in ensuring adequate diversity and difficulty of synthesized instructions. To address these challenges, we propose Self-Foveate, an innovative LLM-driven method for instruction synthesis. This approach introduces a “Micro-Scatter-Macro” multi-level foveation methodology that effectively guides the LLM to deeply excavate fine-grained information embedded in unsupervised text, thereby enhancing both the diversity and difficulty of synthesized instructions. Comprehensive experiments across multiple unsupervised corpora and diverse model architectures validate the effectiveness and superiority of our proposed method. We publicly release our data and codes: https://github.com/Mubuky/Self-Foveate

## introduction large language models (llms), such as gpt-4o (openai, 2024), claude 3.5 sonnet @xcite , and llama 3.1 (meta, 2024), have garnered significant attention due to their exceptional instruction-following capabilities @xcite , with their continuously enhanced problemsolving abilities @xcite being increasingly recognized. a critical component in training such models typically involves fine-tuning with extensive supervised question-answering instruction data @xcite . however, substantial challenges persist ## self-foveate in this section, we will introduce the proposed self-foveate, denoted as f, a multi-level automated method for synthesizing instructions that leverages only unsupervised text without the need for human-annotated samples. self-foveate consists of three levels and a re-synthesis module. formally, consider an original unsupervised text set d, the proposed method self-foveate operates by transforming each element d i from d through a multi-level synthesis path f j . in practice, self-foveate is an operation applied independently to each document d i (d @xmath0. the final generated dataset, d gen , is accumulated from the data subsets generated by all elements: during the synthesis process, the objectives of self-foveate emphasize maximizing the diversity and difficulty of the generated instructions. to ensure optimal functionality, self-foveate incorporates three levels and one module as shown in figure 2 : ## experiment in this section, our experiments focus on three critical research questions: (q1) how effective is self-foveate in enhancing instruction diversity? (q2) how effective is self-foveate in enhancing instruction difficulty? (q3) how effective is self-foveate in improving the model's problemsolving capabilities during instruction fine-tuning? ## conclusion in this paper, we proposed self-foveate, an unsupervised text-based instruction generation method powered by llms, which further explores key challenges in diversity and difficulty. its innovative multi-level foveation and re-synthesis module ensure the acquisition of high-quality, textfaithful instructions from unsupervised text. extensive experiments, including performance evaluations on downstream tasks, demonstrate the effectiveness of self-foveate. our study has obtained many insightful findings, laying the foundation for future research in areas such as instruction difficulty assessment.