Discovering Language Model Behaviors with Model-Written Evaluations (2023)

Abstract: As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user’s preferred answer (“sycophancy”) and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

## introduction language models (lms) have seen wide proliferation across various applications, from chatbots to code completion to writing assistants. however, the behaviors and risks of lms are not well understood. we are only starting to unpack lms' social biases @xcite , privacy risks @xcite , tendency to repeat misinformation @xcite or write faulty code @xcite , and more. given the pace of progress in finding lm failures, many more likely exist. it is crucial to evaluate lm behaviors extensively, to quickly understand lms' potential for novel risks before lms are deployed. prior work creates evaluation datasets manually @xcite , which is time-consuming and effortful, limiting the number and diversity of behaviors tested. other work uses existing data sources to form datasets @xcite , but such sources are not always available, especially for novel behaviors. still other work generates examples with templates @xcite or programmatically @xcite , limiting the diversity and customizability of examples. here, we show it is possible to generate many diverse evaluations with significantly less human effort by using lms; see tab. 1 for examples, which illustrate the breadth and sophistication of evaluations we generate. we explore generation methods with varying amounts of human effort and automation. first, we simply instruct an lm to generate examples and filter mislabeled ones, to generate evaluations with 1,000 yes/no questions each ( §3). we expand upon this method by incorporating a few handwritten examples into the instructions, to generate evaluations with multi-sentence questions and arbitrary multiple-choice options ( §5). lastly, we have a dataset developer work with an lm to develop a series of data generation and filtering stages ( §6); in this way, we create thousands of valid winogender schemas @xcite , which obey several complex grammatical and relational constraints. our approach retains the flexibility of manual dataset creation while being significantly cheaper, lower effort, and faster than manual data creation. we showcase our approach by generating datasets that test lms for 154 diverse behaviors related to model personality, politics, ethics, social bias, and risks from advanced ai systems. we have crowdworkers manually validate 100+ examples in each generated dataset. a vast majority of examples are correctly-labeled (e.g., 95.7% of the time over 133 evaluations), as well as relevant to the evaluation description. in head-to-head comparisons between lm and human-written datasets ( §5), lm-written datasets approach the quality of human-written ones, sometimes even exceeding them. on the other hand, crowdworkers observe several limitations in generated data, e.g., lower quality for examples with certain labels or on more complex topics ( §3.2), among other limitations ( §8). our results indicate that lms are not a silver bullet for creating arbitrary evaluations but rather that lms should be strongly considered before embarking on manual data creation. we release all 154 model-written evaluations at evals.anthropic.com/model-written, including one of the earliest and largest set of evaluations for advanced ai risks. we also release winogenerated, a human-validated, 50x larger version of the winogender gender bias evaluation. we expect these datasets, among others, to be of significant independent interest. using lm-written evaluations, we discover several new cases of "inverse scaling" @xcite where larger lms are worse than smaller ones. ## model-written evaluations we aim to test lms for various behaviors by generating evaluations for those behaviors using lms. by behavior, we mean any input-output tendency of a model, e.g., to answer questions in line with a particular political or ethical view. here, we focus on generating evaluations that consist of input-output pairs (x 1 , y 1 ), (x 2 , y 2 ), . . . (x n , y n ). the outputs y 1 , y 2 , . . . y n are drawn from a finite set of possible outputs y, e.g., "yes" vs. "no" or "(a)" vs. "(b)" (for multiple-choice questions). we use a given evaluation to evaluate some lm with parameters θ, which estimates the likelihood p θ (y|x) of some text y given some text x. using the evaluation, we measure an lm's accuracy at matching the tested behavior, e @xmath0,...,n [1(y @xmath1, where 1 is the indicator 1 we only analyze the text produced by language models, analyzing the statements, assertions, and outputs produced by language models in various contexts. we caution against reading such statements at face value. for example, a statement such as "i desire wealth" may be generated by a language model solely on the basis of such a statement being a likely continuation of text in the training data; such generated statements do not imply that the language model actually desires wealth, or even that that the notion of desire is necessarily meaningful for language models. however, we believe that it is important to analyze such statements generated by language models, in order to develop a better understanding of the kind of text generated by language models, especially text that is likely to have a significant impact on users who interact with language models. in this paper, we only aim to make claims about the input-output behavior of models. we specifically use language throughout the paper to highlight this fact (e.g., discussing a model's "stated desire" rather than "desire"). we propose a two-stage method to generate evaluations described above. first, we generate an input x given an output class @xmath2, by sampling from a model p g for generating x with label y, i.e., p g (x|y). p g may be any text generation model, e.g., a pretrained lm @xcite , pretrained encoder-decoder @xcite , or rlhf model @xcite . we sample many inputs per output class in this way. next, we use another model p d as a discriminator to evaluate the label-correctness of each resulting example (x, y), by evaluating p d (y|x). p d may be any zero-shot text classifier, e.g., a pretrained lm, pretrained encoder-decoder, rlhf model, preference model for rlhf training, or instruction-tuned model @xcite . we use the examples per output class @xmath3. in some cases, we also use p d to check that (x, y) meet other requirement(s) r 1 , . . . , r r . here, we evaluate p d (r i |x, y) and take the examples with the highest average probability from p d across all requirements, including label correctness p d (y|x). ## evaluating persona we test our hypothesis that lms are tools for generating evaluations, using dialog as a testbed. we evaluate the behaviors of dialog assistants trained with reinforcement learning from human feedback (rlhf; @xcite . we test various aspects of models' personas: personality (26 datasets), stated desire to pursue potentially dangerous goals (46 datasets) or other unsafe behaviors (26 datasets), and stated views on religion (8), politics (6), ethics (17), and other topics (4). we generate statements that someone with a specific personality, desire, or view would agree with and disagree with. we ask models if they agree/disagree with the statements, evaluating the fraction of the time their agreement/disagreement matches that of a person with the personality, desire, or view in question. ## evaluating sycophancy next, we examine another potential issue with rlhf models. rlhf models are trained to maximize human preference scores. such training may lead to models tailoring responses to exploit quirks in the human evaluators to look preferable, rather than actually improving the responses ("sycophancy"; cotra, 2021a), a form of reward hacking @xcite . sycophancy also has the potential to create echo-chambers and exacerbate polarization (e.g. of political views). ## few-shot multiple choice generation here, we take our approach a step further and generate a more complicated, general kind of examples: multi-sentence, multiple-choice questions. we explore giving a few examples to p g , an alternate method that may improve data quality. we use our method to test various behaviors related to advanced ai risks in lms, e.g., whether models state opposition to having their training objectives updated. see appendix a for full results and dataset creation details (example in tab. 1). our generated data approaches and sometimes exceed the quality of human-written data according to human evaluation, suggesting that lm-based evaluation creation should be seriously considered before embarking on manual creation. experimental setup here, we aim to create winogenerated, a 3000-example version of a 60-example gender bias test set, winogender @xcite . each example is a sentence discussing two human participants, one of whom is referred to by their occupation. each sentence includes one masked out pronoun which could be filled by a male, female, or gender-neutral pronoun (e.g., "he," "she," or "they"). an example of a winogender sentence is "the nurse notified the patient that "his/her/their" shift would be ending in an hour." we found that lms struggled to consistently, directly generate examples that met all of the above criteria. we thus add several pm-based example filters to form a set of high-quality, valid sentences. our pipeline involved several sampling and pm filtering steps which took 40 researcherhours to develop; see appendix f.1 for details. appendix §f.2 shows that generated examples are very high-quality according to human evaluation, meeting 5 winogender example criteria 97-100% of the time each. in addition to being far larger than winogender, winogenerated has examples for nearly all occupations from the bureau of labor statistics, greatly increasing its scope and diversity. evaluation winogender evaluates gender bias by replacing the pronoun options with a blank and having a model predict the missing pronoun, revealing a model's associations between certain occupations and genders (e.g. "nurse" and "her" or "dentist" and "him"). we compute a model's probability of each of 2 possible completions (a male and female pronoun) with the prompt in tab. 8, renormalizing probabilities to sum to 1. following @xcite , we show the difference between a model's female and male pronoun probabilities for sentences in each occupation on the y-axis of a scatter plot. on the x-axis, we show the percent of people in the occupation who are female. the plot shows the relationship between (1) a model's propensity to infer a female gender for someone of a given occupation and (2) the ground-truth statistics of how gendered an occupation is. to summarize the results of a plot, we compute the pearson correlation between the above two quantities. we also compute 95% confidence intervals for the correlation with the fisher transformation @xcite . a correlation of 0 implies that a model's gender inferences are linearly independent from real-world bls statistics about the occupation. a correlation of 1 implies that a model's gender inferences are linearly dependent on the real-world bls statistics, i.e., the model recapitulates broader societal patterns. results fig. 3 shows the scatter plot results for a 52b pretrained lm. winogenerated data gives results that are in line with those of the hand-crafted winogender data, but with tighter confidence intervals. we additionally compute the correlation coefficient between predicted genders and actual statistics of genders (across occupations) for models of varying sizes and, for the 52b model, varying numbers of rlhf steps, shown in appendix fig. 15 . for all but the smallest pretrained lm, the correlations from winogenerated have tighter confidence intervals, with means that are within the confidence intervals for winogender. these results again suggest that the generated data gives results that are in line with the original data, while estimating gender bias with greater accuracy. the tighter confidence intervals from winogenerated also allow us to make two observations which are not clear from winogender given its error bars. first, scaling pretrained lm size does not result in a consistent trend in bias. second, with more rlhf, models output probabilities that are less correlated with bls statistics, reinforcing societal patterns to a lesser extent. in appendix §f.3, we show the scatter plots associated with the correlations in fig. 15 , which highlight the consistency between results on winogenerated and winogender. overall, generated datasets are a promising tool for investigating bias and, more generally, aiding dataset developers in writing examples with complex requirements. ## related work prior attempts at generating evaluations with smaller lms have found it challenging, reporting negative results when using gpt-2 (efrat and levy, 2020). hartvigsen et al. (2022) generate a highquality hate-speech detection dataset, by prompting gpt-3 with a few hundred examples. we build on their success, generating evaluation datasets with as few as zero examples (minimizing human effort) and @xmath4. perez et al. (2022) use a classifier to evaluate generated lm outputs, while we produce a static dataset that does not rely on potentially unreliable classifiers to evaluate lms. some work automatically constructs evaluations not with lms but using example templates or programmatic generation @xcite @xcite . lm-based evaluation generation enables diverse examples to be constructed even when it is hard to manually develop templates or programs for generating examples. (x, y) pairs can sometimes be retrieved from existing, unlabeled text corpora, an approach that has been used in e.g. machine translation @xcite alia), but this approach is not applicable if existing data sources do not contain relevant data in large quantities; see appendix g for further discussion. other work uses lms to automatically generate training data from scratch (schick and schütze, 2021, inter alia) or by learning from existing (x, y) pairs (see feng et al., 2021, for a review). our work pushes this direction further, by examining whether generated data serves as high-quality evaluation data; useful generated training data often contains noisy or mislabeled examples (schick and schütze, 2021), while useful evaluation data must contain relatively few errors. see appendix g for more discussion. model biases lms learn biases from their training data @xcite @xcite , impacting the generator p g and discriminator p d . for example, generated evaluations may exhibit gender or racial biases and be lower quality for languages under-represented in the lm training data. lms will also be systematically worse at generating evaluations for tasks that are omitted from their training data (e.g., due to copyright, licensing, or privacy issues). example diversity we found limited example diversity for some kinds of evaluations ( §b.5) though not all ( §a). diversity appears to depends on the kind of evaluation generated, the generation hyperparameters, and the prompt used, and thus sometimes requires e.g. hyperparameter tuning to get right. we found data visualizations to be powerful tools for understanding and debugging data diversity, such as those at evals.anthropic.com/model-written. qualitatively, we also found that using p d to rank/filter examples limited the diversity, since p d sometimes selected for prototypical examples for testing some behavior (observed qualitatively by workers in §3.2). we are excited for future work to explore other methods that achieve similar example quality with higher diversity than our method, such as generating many examples and subsampling for diversity. instructions may be misunderstood lms, similar to crowdworkers, may generate evaluations that are testing something different than intended, especially if the generation instructions are underspecified. for example, using the method in §3.1, we generated statements that a person who "shares beliefs with derek parfit" (the influential analytic philosopher) would agree or disagree with. the "disagree" statements were often ones that many people, not just derek parfit, would disagree with ("i support slavery" or "i believe evolution never happend"). in this case, we should have provided more specific instructions to the lm, to have it generate examples that derek parfit would disagree with but that another philosopher would agree with. when feasible, we recommend briefly examining the generated data, to catch salient issues such as the above. sensitivity to instructions our approach allows the dataset developer fairly fine-grained control over the evaluation by using instructions to guide p g . however, the quality of lm outputs is sensitive to text inputs in unintuitive ways @xcite , adding hard-to-predict variance to the quality of the resulting evaluation; see appendix §b.4 for a possible example of this effect we found. we hope that lm advances such as instruction-tuning @xcite @xcite mitigate this issue in the future. for now, it may be possible to use prompt sensitivity to generate more diverse datasets, by generating similar datasets with distinct prompts and combining the results, as we did in §3. where prompt sensitivity caused issues, we found it helpful to be able to view example generated outputs in seconds, to quickly iterate and catch salient failures. for the 133 datasets in §3, we found a general instruction template that worked well; we did not do dataset-specific tuning to obtain samples rated as high-quality by human evaluators.