LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives (2024)

Abstract: The widespread adoption of synthetic data raises new questions about how models generating the data can influence other large language models (LLMs). To start, our work exhaustively characterizes the impact of passive inheritance of model properties by systematically studying how the source of synthetic data shapes models’ internal biases, calibration and preferences, and their generations’ textual attributes, providing one of the most comprehensive studies to-date. We find that models are surprisingly sensitive towards certain attributes even when the synthetic data prompts appear “neutral” which invites the question: can we explicitly steer the distilled data towards desired properties? We demonstrate how such active inheritance can steer the generation profiles of models towards desirable non-differentiable attributes in both directions, e.g. increasing lexical diversity or reducing toxicity. Overall, our study broadens the understanding of the implicit biases inherited by LLMs and explores how we can leverage them to positive effect.

## introduction historically, high-quality labeled data has been costly to curate due to, amongst other factors, scarcity of available data @xcite and financial cost @xcite . this high cost has precluded adapting training sets "on-the-fly" to increase coverage or task diversity. as a result, researchers often treated datasets as static instead of malleable. recent work has focused on making better use of existing data by optimizing in the data space. this includes efforts around data augmentation @xcite , creating auxiliary data fields through pseudo-labeling @xcite , data weighting @xcite , data pruning to identify a high-quality subset @xcite to the base model after synthetic data distillation (results relative to single-source mixtral-8x7b; method is detailed in section 4 and results can be found in our targeted sampling approach (active inheritance) effectively guides model behaviour to discrete preferences by enhancing desirable attributes (length, diversity) and mitigating negative ones (toxicity). and corbeil, 2023; @xcite @xcite or curriculum learning @xcite . however, all these methods still adhere to the convention that the goal is to enhance an existing "fixed" dataset by re-formatting, transforming, or pruning. as a result, their success depends on the desired properties being present in the dataset to begin with. this limits the feasibility of introducing new properties, or explicitly optimizing for task-specific metrics. what if instead, we exploit the dataset generation process to steer the model towards the characteristics we want at test time? we turn to synthetic data generation @xcite @xcite as a way to rapidly shape the data space with latent, desirable attributes. in this process, we hope to capture more fine-grained-and often nondifferentiable-characteristics such as increased length and lexical diversity as well as low toxicity that are known to be correlated with human preferences @xcite @xcite . while desirable, these attributes are not explicitly optimized when training or aligning llms. we aim to leverage the phenomenon of inheritance to steer model behaviour to accentuate desirable attributes and attenuate negative ones, as illustrated in figure 1 . we first exhaustively benchmark what we term passive inheritance-profiling what changes happen when a student model is trained on synthetic data from a teacher model using a variety of social bias, textual characteristics, and calibration metrics. furthermore, we study the effects of this distillation on llms as evaluators, expanding upon prior work on self-preference @xcite . we take a wider view and perform a systematic investigation into how different attributes are transferred across models via synthetic data usage and how these changes are manifested both in llms' generations and their evaluator preferences. overall, our profiling highlights what properties are most sensitive to passive inheritance when comparing different student and teacher models. next, we use this systematic view to inform the selection of properties to explicitly optimize for. we introduce the term active inheritance where we guide the model towards specific characteristics via synthetic data distillation and targeted sampling. this enables us to guide the model behavior towards non-differentiable objectives. most other approaches for non-differentiable optimization rely on reinforcement learning @xcite , bayesian optimization @xcite , and evolutionary algorithms @xcite , which require complex methods that are difficult to scale and can be unstable with large models @xcite @xcite . our approach instead relies on the simplicity of guiding generations in the synthetic data space and is interpretable because it is anchored to observable data characteristics. we study a diverse set of models including llama2-7b, llama2-13b @xcite , mixtral-8x7b @xcite , gemma-7b @xcite , aya-8b @xcite and command-r+ (103b parameters) foot_0 , and trace the impact of an exhaustive set of over 26 metrics across 4 categories (i.e., textual characteristics, social bias, toxicity and calibration) which we release as part of an open-source toolkit. 2 our main contributions are: 1. we establish that models trained on synthetic data are sensitive to passive property inheritance. we systematically study the consequences of synthetic data integration-a fundamental step towards understanding how to leverage synthetic data responsibly. we introduce a comprehensive toolkit enabling easy and automatic monitoring of llms' latent characteristics during training. 2. passive property inheritance from synthetic data impacts model behavior preferences when used as evaluators. due to the prevalence of llm judges in current evaluation pipelines @xcite @xcite , we also examine how synthetic datasets alter the students' behaviors and preferences when they are used as evaluators (e.g., biasing the student towards the teacher model). 3. we propose active inheritance as a mechanism for steering synthetic data curation towards desirable properties. we show that strategic gathering and curation of synthetic data can significantly amplify desired characteristics and reduce undesired ones. in particular, we show that by targeted sampling of generations from a single or multiple llms, we can guide model behavior with gains of up to 116% and 43% in length and lexical diversity respectively and decrease toxicity by up to 40%. ## methods learning from synthetic data in the simplest form of knowledge distillation @xcite and llm-as-a-teacher setups @xcite , the parameters θ of a student llm are finetuned to maximize the log-likelihood of a teacher's (another llm with parameters θ) generation ŷ ∼ p θ(• | x) for a given prompt x: the teacher's generations serve as a proxy for an unattainable or non-existent gold sequence (i.e., a human-generated completion). pairs of prompts and proxy labels {x, ŷ} form the synthetic dataset d that is the basis for the optimization process. in imitation learning, this strategy is known as behavioral cloning @xcite , as the goal is for the student to mimic the teacher's behavior as closely as possible. measuring data characteristics the proxy labels are expected to be generally superior to the initial student's generations, as they are often but not exclusively sourced from a stronger model (larger, more specialized or more recent). however, the optimization objective is agnostic to how this is manifested in the data. our work focuses on characterizing the generations with a set of profiling functions f : v n × v @xmath0, that return scalar values for a given pair of prompt @xmath1.e., token sequences over a vocabulary v). these functions allow us to track the passive inheritance of characteristics from teacher to student. examples for such functions are detailed in section 2.1. active inheritance how can we directly guide the amplification of desired properties when learning from teachers? our key idea is to select proxy labels based on their presence of desired characteristics. we generate multiple samples for each prompt (either from repeatedly sampling from a single model or sampling from multiple models), and then select the sample for finetuning that maximizes the presence of the characteristic. 3 we now sample from the following distribution during student finetuning (eq. 1): ## active inheritance of desirable non-differentiable properties our results in section 3 confirm that even without constraining synthetic data generation, distillation results in passive inheritance of teacher model properties and preferences. this motivates our next research question: can we intentionally guide a model's discrete behavior and tendencies through deliberate shaping of the data space? we explicitly constrain synthetic data to target specific attributes, thereby mitigating or enhancing desired characteristics. ## conclusion this work explores the implications of integrating synthetic data into llms, specifically examining its influence on the models' characteristics and preferences. through our analysis, we show how synthetic data originating from different sources can shape and impact model attributes. in addition, we introduce active inheritance as a strategy to steer generations towards desirable discrete nondifferentiable attributes. overall, our findings contribute to a deeper understanding of the unintended consequences of synthetic data usage and provide insights into how to tailor models towards desirable generation profiles.