Efficiently Selecting Response Generation Strategies for Synthetic Data Construction by Self-Aligned Perplexity (2025)

Abstract: Fine-tuning large language models (LLMs) typically relies on producing large sets of input-output pairs. Yet for a given question, there can be many valid outputs. In practice, these outputs are often derived by distilling knowledge from teacher models, and they can vary depending on the specific teacher model or prompting strategy employed.Recent findings show that how these training outputs are generated can significantly affect the performance of the fine-tuned model, raising an important question: how do we pick the best data generation method from among numerous possibilities? Rather than exhaustively training and evaluating on each candidate, this paper proposes a scalable approximate method that assesses a small subset of generated data to estimate its suitability for a specific target LLM. Our central idea is that effective outputs should be familiar to the target LLM. While previous work measures familiarity with perplexity, we find that perplexity might be suboptimal in characterizing “familiarity” through empirical analyses and practical observations. To address this, we introduce self-aligned perplexity, a novel metric capturing how closely candidate outputs adhere to the target LLM’s own style and reasoning patterns. In this way, we can identify the most effective generation strategy on a small sample, then apply it to produce the complete training set. We demonstrate that training on data generated by the chosen method yields significant improvements across diverse reasoning-focused benchmarks, particularly in cases where different candidate methods lead to highly divergent training outcomes.

## introduction when instruction-tuning an llm, training data consists of question-response pairs, where multiple valid responses can be generated for the same input. previous studies @xcite show that datasets with identical input questions but different responses can lead to varied learning outcomes, even when responses contain similar levels of detail. this raises a key question: how can we construct responses that are most effective for the target llm? prior research has explored improving responses by adding details or rationales, such as structuring ground truth step by step @xcite , incorporating rationales, or enriching responses with additional information @xcite @xcite . however, recent studies @xcite suggest that more details or converting responses to step by step style do not always improve performance and that alignment with the llm's linguistic style is crucial. in our experiment, we observe that no single response generation strategy works universally across tasks. thus, we need to create a method to find out the most effective way to generate responses for each task, rather than a single method for all tasks. the concurrent works @xcite attempt to predict the effectiveness of response generation methods by evaluating the entire training dataset. they generate full training datasets using each method and then estimate training effectiveness based on scores computed via algorithms or reward models. however, these approaches are computationally expensive and not scalable. however, can we predict the effectiveness of each data generation methods efficiently? we observe an interesting phenomenon that each response generation method produces responses with a consistent style, meaning that a small subset of generated examples can effectively represent the entire dataset. based on this assumption, we propose an efficient ranking pipeline that evaluates a limited number of samples (e.g., 50) to assess the performance of each response generation strategy. this approach uses an alignment estimation function to assign scores to each strategy, enabling us to identify the best-performing method without the need for a full-dataset evaluation. previous research @xcite used perplexity to measure a model's familiarity with candidate question-answer pairs, proposing that lowerperplexity responses for the same input tend to yield better training performance. however, we found several cases where perplexity-based filtering was ineffective. for instance, responses structured in a step-by-step or redundant style often exhibit low perplexity but do not necessarily improve training outcomes. while some candidate responses (e.g., step-by-step or redundant ones) may achieve low perplexity under the target llm, the model itself rarely generates such responses when producing answers freely. this suggests that low perplexity does not always indicate alignment with the model's inherent reasoning style. these findings suggest that perplexity can be "hacked" by response style. thus, traditional perplexity alone is insufficient for selecting the best response generation strategy. to address this, we propose self-aligned perplexity, a refined metric for measuring a model's familiarity with target responses. the key idea is that a model is most familiar with the data it generates itself. leveraging this, we modify perplexity computation by incorporating model-generated responses as in-context examples. specifically, we first have the model produce initial responses, which is then appended to the question as in-context examples. a prompt enforce the model to pay attention to these examples when computing perplexity, thereby altering the probability estimation of the candidate response. if the target response deviates significantly from the model's own generated response-the one it is most familiar with-the model assigns it a lower probability, increasing its perplexity. our experiments show that self-aligned perplexity outperforms traditional perplexity in selecting effective data generation strategies. in our experiments, we observe a strong correlation between the proposed indicator and the ranking of training dataset performance. furthermore, we construct a pool of answer generation strategies and demonstrate that applying our selection criterion leads to significant performance gains compared to the baselines-especially in scenarios where different data-generation methods produce highly divergent outcomes 2 related works there has been extensive research into what types of data yield the best training outcomes for large language models (llms). previous studies have identified several factors that positively influence model training, such as adding complexity @xcite , adding details @xcite @xcite , adding diversity @xcite , augmenting ground-truth answers in a step-by-step manner @xcite @xcite @xcite , and ensuring correctness @xcite . however, in practice, these metrics are challenging to measure for a given dataset, making it difficult to determine the quality of training data based on these criteria. @xcite found that familiarity, measured by perplexity, significantly impacts model training. perplexity has been widely used for different purpose in prior research. perplexity has been used to select prompts @xcite , showing that prompts with lower perplexity generally lead to better performance in question-answering tasks. it has also been used for selecting pretraining datasets @xcite , detecting ai-generated content @xcite , and selecting instruction-tuning data from a database @xcite . @xcite modify the perplexity score and propose "ifd" (instruction following difficulty), which is used to select a small pool of challenging data from the original dataset for efficient training. researchers hypothesize that higher perplexity indicates more challenging data, which can be beneficial for teaching llms new knowledge. in addition, perplexity or confidence-based curricula have been explored for nmt @xcite and general text generation @xcite , where harder (high-perplexity) data are introduced progressively to improve sample efficiency. unlike these studies, our focus is on identifying the best strategy to generate target responses (y) for a given input (x), rather than selecting difficult (x, y) pairs for training language models. recent efforts have begun to ask which teacher model produces the most useful synthetic targets. @xcite introduce a compatibility-adjusted reward (car) and judge its quality by the spearman correlation between car scores and downstream accuracy on two instruction-following datasets, each evaluated with a single meta-prompt. @xcite study nine datasets spanning mathematics, coding, and general instructions; they correlate several corpus statistics with training gains and combine them with principal-component analysis to rank teacher models. our study differs in four key respects. first, we estimate a strategy's quality from only a small sample of its outputs, making synthetic data generation and evaluation far more affordable. second, we target accuracy improvement, not just rank correlation. third, we experiment on a much broader benchmark: 17 diverse tasks plus 6 plan-bench planning tasks. fourth, we evaluate our method on datasets generated using diverse meta-prompts, explicitly accounting for prompt variability. ## method this paper aims to efficiently select the most effective answer generation strategy for fine-tuning a target llm. in what follows, we first present the problem setup, then detail our proposed selfaligned perplexity metric for scoring the outputs from each candidate strategy. ## benchmark construction in this section, we show how we use different strategies (distinct prompts and teacher llms) in generating high-quality responses with different styles. ## experiment in this section, we treat each generation strategy from section 4.3, and response-selection metrics from the related work section, as baselines. we then compare the average training outcomes of our method against these baselines across all tasks. there are two benchmark sets, detailed in section 4.2. the first is the main-experiment corpus, which covers a diverse range of tasks and serves as the primary benchmark for evaluating both the general ranking ability of our metric and the average performance gains achievable by our method. since our goal is efficient data selection, we evaluate each metric using only a small subset of the training data. for each method, we repeat the process three times, each time selecting a different subset of size @xmath0, and report the average performance across these runs. for example, one run may use the first 50 samples, another the second 50, and so on. the final result is computed as the average of these three evaluations. ## conclusions in this paper, we present a novel and scalable approach for selecting the optimal responsegeneration strategy to train large language models. we introduce a new metric, self-aligned perplexity, which more effectively evaluates the alignment between a target model and its response options compared to traditional perplexity. we demonstrate that choosing the optimal generation strategy based on self-aligned perplexity leads to substantial improvements in model performance, particularly on tasks with high performance variance. we hope our work will inspire researchers who use perplexity as a downstream metric or who wish to build the most effective instruction tuning datasets.