Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts (2025)

Abstract: We present Autonomous Data Selection (AutoDS), a method that leverages base language models as zero-shot “generative classifiers” to automatically curate high-quality mathematical texts. Unlike prior approaches that require human annotations or training a dedicated data filter, AutoDS relies solely on a model’s logits to determine whether a given passage is mathematically informative and educational. By integrating AutoDS into a continual pretraining pipeline, we substantially boost downstream performance on challenging math benchmarks (MATH, GSM8K, and BBH) while using far fewer tokens than previous methods. Empirically, our approach achieves roughly a twofold improvement in pretraining token efficiency over strong baselines, underscoring the potential of self-directed data selection in enhancing mathematical reasoning. We will release our curated dataset to facilitate future research in automated domain-specific data curation.

## introduction language models (lms) have witnessed tremendous advancements, becoming increasingly adept at natural language understanding, generation, and reasoning @xcite @xcite @xcite . yet, integrating domain-specific knowledge into these models remains a critical and challenging frontier @xcite . mathematical reasoning, in particular, demands specialized expertise: texts often contain symbolic formulas, multi-step derivations, and rigorous proof structures that differ considerably from conventional language tasks @xcite @xcite . despite the growing enthusiasm for building lms with robust mathematical proficiency, the field continues to face a scarcity of well-curated and highquality mathematical corpora, underscoring the urgent need for innovative approaches to create and refine domain-specific training data. recent efforts have begun to address this gap. for instance, @xcite and @xcite demonstrated the utility of large lms (e.g., gpt-4) to appraise the educational value of code snippets in the stack dataset @xcite , subsequently training a traditional classifier (e.g., random forest) for data filtering. while these approaches represent a pivotal step toward more judicious data curation, they typically produce only discrete labels (e.g., "good" vs. "bad"), discarding the finer granularity of data quality. in mathematical contexts, subtle nuances matter: a dataset entry with an "educational value" of 0.95 should arguably be treated differently from one at 0.001. relying solely on binary classification can thus limit the efficiency and precision of the training pipeline. a promising alternative is to assign continuous real-valued scores to each data point, thereby enabling the model to focus selectively on the most informative texts. however, constructing such a continuous scoring system poses nontrivial challenges. large language models often struggle with generating reliable numerical values or sampling consistently from intricate distributions @xcite . drawing inspiration from the direct preference optimization (dpo) framework @xcite , we propose a simpler yet effective solution: leveraging the model's own logits associated with targeted tokens (e.g., "yes" vs. "no") to produce a quantitative score function. this approach avoids costly labeling efforts and bypasses the need for training an additional classifier on human-annotated data. concretely, we introduce autonomous data selection (autods), which uses zero-shot metaprompts to evaluate the quality of mathematical texts for continual pretraining. instead of relying on aligned or fine-tuned models, we take a strong base model and prompt it with two yes/no questions assessing (1) the level of "mathematical intelligence" in the text, and (2) its utility for future math learning. from the resulting logits on "yes" and "no," we compute a single real-valued lm-score that captures the text's educational value. this enables a more fine-grained assessment than binary filtering approaches @xcite , thus amplifying token efficiency by selectively training on the most instructive samples. another distinguishing factor of our method is its ability to autonomously curate data: no separate human-annotated corpus or reward model is needed. techniques like supervised fine-tuning (sft) @xcite , reinforcement learning from human feedback (rlhf) @xcite , or specialized preference modeling @xcite are not required. by directly applying a softmax-based score on the base model's logits, autods orchestrates a form of active, self-directed learning, where the model itself identifies and harnesses the best materials for continual pretraining. this paves the way for a more dynamic and scalable data selection pipeline, especially relevant for highly specialized fields like mathematics. empirically, we show that continually pretraining language models on this auto-curated dataset yields substantial gains on mathematics benchmarks, such as math @xcite , gsm8k @xcite , and big-bench hard (bbh) @xcite . remarkably, these improvements come with far fewer tokens compared to previous continual pretraining works, effectively boosting training efficiency by roughly a factor of two. figure 1 previews these performance trends on the mistral-7b model @xcite , underscoring the efficacy of our data selection method. our key contributions are three-fold: ## language models as zero-shot generative classifiers recent advances in large language models (llms) have demonstrated remarkable potential for complex reasoning and decision-making @xcite . building on these capabilities, we propose leveraging base language models in a zero-shot fashion to verify whether candidate documents possess the mathematical rigor and educational utility necessary for continual pretraining. this approach goes beyond conventional few-shot paradigms @xcite , which require task-specific prompt engineering or model fine-tuning, by directly harnessing the llms' inherent capacity to assess textual content without reliance on human annotations. ## autonomous data selection with language models building on the zero-shot verification approach outlined in section 2, we apply our lm-score-based data selection pipeline to three principal sources of mathematical texts: these sources cover a wide range of mathematical domains and difficulty levels, making them wellsuited for continual pretraining. experiment details. we process a total of 11.26 m documents, amounting to over 200 gb of data. following the methodology presented in section 3, we obtain an lm-score for each document using the qwen-72b base language model @xcite and retain documents scoring above specified thresholds. we employ the vllm inference framework @xcite on nodes with a100-80g and a800-80g gpus. the entire filtering procedure required roughly 750 gpu hours on 4 a100-80g gpus (i.e., 3000 gpu hours total), including both loading and inference. by contrast, expert manual annotation of 11.26 m documents (at around @xmath010 m. using standard commercial cloud pricing for gpu compute (@xmath110k, drastically reducing labeling cost while avoiding the pitfalls of rule-based or purely keyword-based filtering. visualization of data composition examining how the selected data are distributed across different websites and content types provides insight into the quality and variety of the resulting corpus. in figure 4 , we plot a tree map showing the top 30 domains that scored in two different lm-score ranges. as indicated, *.stackexchange.com contributes a substantial share of high-scoring examples, many of which are not yet fully leveraged in other opensource math corpora @xcite . figure 5 offers a more granular breakdown of the highest-frequency domains and the proportion of documents falling into high-scoring bins (e.g., 0.75-1.00). we observe that many math-intensive websites such as math.stackexchange.com and mathhelpforum.com have a particularly large share of high-scoring data, underscoring their suitability for enhancing advanced mathematical language modeling. ## experiments in this section, we empirically assess the effectiveness of our proposed autods method in enhancing mathematical reasoning through continual pretraining. we demonstrate that our approach substantially improves performance on several mathfocused tasks while using significantly fewer tokens than previous works. we further compare autods against existing baselines and evaluate its broader impact on general reasoning tasks. ## conclusion we introduced autods, an autonomous data selection framework that transforms base language models into zero-shot "generative classifiers" for filtering mathematical texts. by relying solely on model logits to assign real-valued scores, our approach avoids the need for human-annotated labels and enables more fine-grained curation than conventional binary classification methods. through extensive evaluations, we found that continually pretraining language models on this self-selected corpus markedly enhances mathematical reasoning skills while consuming significantly fewer tokens. moreover, the improvements on math, gsm8k, and bbh underscore the effectiveness of autods in identifying and prioritizing instructive content. looking ahead, we plan to extend autods to broader domains to further explore how self-supervised data selection can advance specialized nlp tasks. we hope that making our au-tomathtext dataset publicly available will foster further research on scalable and autonomous data selection approaches for domain-specific training.