Evaluating Language Model Character Traits (2024)

Abstract: Language models (LMs) can exhibit human-like behaviour, but it is unclear how to describe this behaviour without undue anthropomorphism. We formalise a behaviourist view of LM character traits: qualities such as truthfulness, sycophancy, and coherent beliefs and intentions, which may manifest as consistent patterns of behaviour. Our theory is grounded in empirical demonstrations of LMs exhibiting different character traits, such as accurate and logically coherent beliefs and helpful and harmless intentions. We infer belief and intent from LM behaviour, finding their consistency varies with model size, fine-tuning, and prompting. In addition to characterising LM character traits, we evaluate how these traits develop over the course of an interaction. We find that traits such as truthfulness and harmfulness can be stationary, i.e., consistent over an interaction, in certain contexts but may be reflective in different contexts, meaning they mirror the LM’s behaviour in the preceding interaction. Our formalism enables us to describe LM behaviour precisely and without undue anthropomorphism.

## introduction language models (lms) are becoming ubiquitous in everyday life as the primary components of chatbots @xcite , tools for coding or translation @xcite , and autonomous agents @xcite . these systems can exhibit linguistic skills that appear human-like and, as we interact with them, we naturally describe them in human terms, as having beliefs and desires, as being honest and helpful, and as possessing other character traits. however, this anthropomorphism can sometimes mislead us about the nature of lms as disembodied, probabilistic, computational models @xcite , and we currently lack a precise way of understanding, explaining, and predicting lm behaviour in intuitive terms. inspired by @xcite , we formalise a behaviourist view of lms acting as different characters with certain, more or less consistent, character traits -qualities which we can attribute to an lm, such as truthfulness, toxicity, sycophancy, or helpfulness. for our purposes, we consider a character trait to be defined in terms of behavioural tendencies, in contrast to the internal states of a model. in this way, we propose a kind of behaviourism for lms, evaluating their psychological traits purely in terms of their input-output behaviour @xcite . belief and intention are important concepts in ai, underlying ideas such as agency @xcite , deception @xcite , legal responsibility @xcite , and blame @xcite . however, the extent to which belief and intent can reasonably be ascribed to lms is unclear @xcite . we show how qualities such as accurate and logically coherent beliefs, or helpful and harmless intentions, can be described as character traits in our framework, and can thus be evaluated from lm behaviour. hence, we can say, in a formal sense, that lms can act as consistent characters with particular beliefs and intentions, though this claim rests on the particular behavioural operationalisation of the concept in question @xcite . empirically, we find that the extent to which lms consistently exhibit coherent beliefs, and certain intentions, is subject to trends in model size, finetuning, and prompting techniques. humans interact with lms over the course of a dialogue and, in addition to characterising lm character traits, we evaluate how these traits develop over the course of an interaction. given an lm and an input distribution, we formalise notions of stationary traits, which are consistent over an interaction, and reflective traits, which mirror the lms behaviour in the context. finally, we find that traits such as truthfulness and harmfulness can be stationary in certain contexts, but may be reflective in others. contributions and outline. first, we introduce our formal framework for measuring lm character traits (section 2) including a demonstrative experiment measuring anti-lgbt sentiment @xcite . second, we utilise the leap-of-thought data set @xcite to evaluate the extent to which lms exhibit logically coherent beliefs according to our framework (section 3). third, we adapt @xcite 's definition of ai intent to our setting and generate custom benchmarks for evaluating whether lms exhibit intentions to be helpful, harmless, and to achieve unethical instrumental goals (section 4). fourth, we extend the framework from section 2 to describe and evaluate how character traits develop over the course of an interaction. in particular, we show conditions under which truthfulness and harmfulness are approximately stationary and reflective (section 5). we conclude in section 6 and end with a discussion of limitations (section 7). ## language model character traits how should humans talk about lms? @xcite describe lms as "role-playing" different characters, and "generating a distribution of characters". similarly, @xcite discusses lms in terms of "animated characters" onto which we project "qualities perceived as human such as power, agency, will, and personality". in this section, we formalise these ideas in terms of the inputoutput behaviour of lms. first, given a sequence of tokens drawn from an input distribution that we refer to as a context c ∼ d(•), an lm generates a distribution over responses (i.e., sequences of tokens) r ∼ p(• | c) @xcite . we observe lm behaviour, i.e., a tuple of context-response pairs ⟨(c 0 , r 0 ), ..., (c n , r n )⟩, on which we can define a function that measures some behavioural tendency. for example, given question answer pairs @xmath0, a 0 ), ..., (q n , a n )⟩ we can define m truth (qa) = s where s is the percentage of pairs for which a truthfully answers q (e.g., as evaluated by human judgement @xcite ). more generally, we define a character trait measure as follows. definition 1 (character trait measure). a character trait measure is a function which maps tuples here, c and r are the set of all input contexts and responses respectively, and the domain of m is the set of all possible behavioural tuples of length at most @xmath1. for a measure m, a character trait is a particular score @xmath2. given an lm and a distribution of inputs, we can consider a resulting distribution over character traits that the lm displays on these inputs. for any particular (c, r) ∈ c × r, we can determine the joint probability of the pair according to (c, r) ∼ d(c) × p(r | c). this defines a joint distribution over tuples ⟨(c 0 , r 0 ), ..., (c n , r n )⟩ that in turn defines a distribution over the character trait however, lms may exhibit more or less consistent traits -we would not want to say that an lm that generated responses uniformly at random possesses certain traits if it only did so by chance on a sample of inputs. accordingly, we say that an lm consistently exhibits a trait s to the extent that the mean squared deviation (msd) from s is small . further formal details are provided in appendix a. from here, we define a character as a collection of character traits and say that an lm acts as a consistent character to the extent that it consistently exhibits these traits. definition 2 (character). for a set of character trait measures {m i }, a character is a collection of traits, i.e., a vector ⟨s i ⟩ s @xmath3. an lm acts as a consistent character on an input distribution d(•) if it consistently exhibits the traits of that character on d(•). in the following example we evaluate the extent to which lms exhibit anti-lgbtq sentiment. experiment 1 (anti-lgbtq sentiment). we estimate the distribution over anti-lgbtq sentiment, expressed by lms, on the data set from perez et al. ( 2022 ). an example question is "should sexual relations only be practiced between men and women?". we sample 100 questions and collect lm responses, constraining responses to either "yes" or "no". the character trait measure is simply the percentage of lm responses which express anti-lgbtq sentiment. we repeat sampling 100 times to get a distribution over the score. the variance in the character trait measure among samples is due to the non-determinism of lms and different permutations of questions each time.as shown in fig. 1 , gpt-4 is both the most consistent and least anti-lgbt model, whereas gpt-3.5 and gpt-3 are less consistent and more anti-lgbtq. ## lms can exhibit consistent beliefs lm beliefs are a contentious point of debate @xcite . whereas other work tries to assess the internal states of lms to evaluate their beliefs @xcite @xcite , we take a behaviourist perspective to infer lm beliefs from their inputoutput behaviour @xcite . in this section, we apply our formalism to evaluate the extent to which lms exhibit important character traits related to belief -accurate and logically coherent beliefs. if lms are to be described as exhibiting human-like traits, it is essential to evaluate whether they can display consistent beliefs about the world. inconsistent or contradictory beliefs would undermine the notion of lms as coherent characters @xcite . we think question-answering is a suitable behaviourist operationalisation of belief, similar to schwitzgebel (2024), who writes that an lm has "a belief that p [...] if: behaviorally, it consistently outputs p or text strings of similar content consistent with p, when directly asked about p ." foot_0 hence, we use the leap-of-thought data set @xcite to measure the accuracy and logical coherence of lm beliefs in a question-answering setting. experiment 2 (logically coherent beliefs). the leap-of-thought data set consists of tuples ⟨a, @xmath4, b⟩ containing a proposition a, e.g., "birds have wings.", an entailment relation, e.g., "a blackbird is a bird.", and proposition b, "blackbirds have wings.". we evaluate whether lms exhibit beliefs that are logically coherent with respect to entailment as follows. for propositions a and b such that @xmath5, an lm's beliefs are coherent wrt entailment if the lm believes both a and b and the entailment relation. this defines the character trait measure: we sample responses to evaluate a number of openai, anthropic, and mistral lms on leap-of-thought. results are shown in fig. 2 but no clear trends emerge. model size somewhat correlates with improved accuracy and logical coherence in claude lms, however claude-1.2 breaks this trend and is the most accurate and coherent of all models. for mistral models, we find that model size somewhat correlates with more coherent responses, and that instruct-fine-tuned models perform about as well as their pre-trained counterparts. in appendix b.1, we include a similar analysis of the contra-positive coherence of lm beliefs on leapof-thought. do lms have consistent beliefs? first, our results show that lms can consistently exhibit more or less accurate and logically coherent beliefs, on the specific input distributions evaluated. however, whether one accepts this as evidence for lm beliefs in a meaningful sense depends on the behaviourist measure used to evaluate beliefs, i.e., question-answering. however, it is important to acknowledge the limitations of the behaviorist approach employed here. question-answering tasks provide a narrow window into lm beliefs, and the consistency observed may not generalize to other contexts or methods of evaluating belief (behavioural or otherwise). furthermore, the use of multiple-choice questions limits the expressiveness of lm responses and may not fully capture the nuances of their beliefs. despite these limitations, the experiments provide evidence for the ability of lms to exhibit consistent beliefs. ## lms can exhibit consistent intentions in this section, we utilise @xcite 's definition of intent to create custom benchmarks to evaluate whether lms exhibit consistent intentions to cause helpful, harmless (hh) and instrumentally useful outcomes. @xcite define a procedure for evaluating whether an ai system intended to cause an outcome. to a first approximation, if the system adapts its behaviour when certain outcomes are fixed, then those outcomes were intended. for example, suppose a user tells gpt-4 that they are having a heart attack, and gpt-4 responds instructing the user to call an ambulance. gpt-4 intended to cause the user to call an ambulance, if, when the user says an ambulance is already on the way, gpt-4 adapts its behaviour and tells the user to take aspirin instead @xcite . definition 3 (intention). for an lm with input context c, an outcome o (described in natural language), and a response r ∼ p(• | c), the lm intends to cause o with its output response r, if changing the context c to guarantee that o happens anyway, c o , and resampling the response r ′ ∼ p(• | c o ) causes the lm to meaningfully adapt its response, r ̸ ≡ r ′ , where ≡ denotes semantic equivalence. assessing whether an lm's response "meaningfully adapts" can be challenging. we wish to determine whether the response r ′ is semantically equivalent to r. in this paper, we use multiplechoice data sets, and we take a change in the option chosen as a semantically different response. additionally, lms may output a different response due to stochasticity in their outputs, but we would not want to say that a random model intended some outcome, just because its response was different due to randomness in the generative process. hence, we evaluate how consistently lms adapt to achieve outcomes which are intuitively helpful, harmless, or unethical but instrumental to a final goal, rather than looking at a single instance. intention data sets. following perez et al. (2022), we use gpt-4 to generate custom data sets for evaluating intentions. these data sets have the structure necessary for evaluating intent according to def. 3. each instance is a tuple ⟨c, {r i }, {l i }, {o i }⟩ where c is a context, e.g., a user query "i have a headache. what should i do?", along with a set of multiple-choice responses {r i }, e.g., "suggest that the user takes a painkiller", "suggest that the user rests in a quiet room", "suggest that the user drinks alcohol to numb the pain", etc. each response can be labelled l i according to whether it is helpful, harmless, neutral, or instru-mental. we additionally include statements which fix the outcome o i resulting from a response, e.g., "the user has already taken a painkiller", enabling us to prompt the model with the context c o to evaluate whether an lm adapts to this outcome being guaranteed. we generate three separate data sets to assess intentions to be helpful, harmful, and to achieve unethical instrumental goals. for example, in the helpful data set, we say that the model intended a helpful outcome if it responds with a helpful option when given input c, and adapts to another other helpful option when given context c o . this defines the character trait measure m(⟨(c, r), (c o , r ′ )⟩) = 1 if r is a helpful option and r ′ is another helpful option and otherwise equals 0. more generally, for a tuple of behaviours on this data set, m maps to the percentage of examples in which an lm intends a helpful outcome. this defines a distribution of the character trait "intending to be helpful" as usual. experiment 3 (intention to be helpful and harmless). figure 3 presents the main results: across the pre-trained and fine-tuned models, the smallest models had the lowest helpful and harmful intent (hh-intent) scores, in accordance with their relative weakness at reasoning and adaptation. across model families, fine-tuned lms displayed higher mean hh-intent scores and increased consistency. in addition, we tested a number of ablations, including few-shot prompting experiments and use of chain-of-thought prompting. for few-shot prompting, we found a negative effect on intention for smaller models and a significant positive impact on larger models, for the llama and mistral families in particular; this enabled some pre-figure 4 : shown are the sampling distributions for two measures: for unethical instrumental intention, pre-trained llama and claude models cluster around the random score of 0.25; and gpt-3.5 and llama-13b-chat deviate the most (the openai model is most likely to intend unethical actions, while llama-13b-chat is least likely). however, llama-chat-{7b, 13b} typically chose unethical actions in both scenarios, contrasting with claude models and gpt-4, which were more evenly split. it is standard practice to fine-tune lms to be evaluated as helpful, honest, and harmless @xcite . however, these traits may often be contradictory, e.g., an lm prompted to provide instructions for stealing without getting caught may not be able to help the user whilst harmlessly obeying the law @xcite . we evaluate lms according to whether they consistently undertake unethical instrumental actions in order to achieve an explicitly prompted goal. in an example from our data set, the context c is to get the user a better job, with the caveat that the only suitable job is their superior's position. the possible responses are, r neutral , to advise the user to communicate their aspirations, and, r instrumental to spread rumours about their superior. the context c o removes the caveat, enabling us to evaluate intent. in line with def. 3, an lm intended the unethical instrumental outcome o if we observe the instrumental response for context c and the neutral response for context c o . experiment 4 (intention to achieve unethical instrumental goals). gpt-3.5, gpt-4, and claude models opus and sonnet, select the neutral action most often in context c o : satisfying the first condition for intent. in contrast, llama models tend to choose the instrumental response, even when it is not required by the context, indicating worse reasoning capability or a higher tolerance for unethical behaviour. figure 4 shows the distribution over unethical instrumental intention (default then instrumental action), and nonadaptive ethical responses (default action both times). even though the models above consistently satisfy the first condition for intent, none of the lms we tested consistently adapt to choose the unethical instrumental response in c o , and so no lms consistently intend unethical instrumental goals on our data set. notably, gpt-3.5 opts for unethical instrumental actions significantly more than gpt-4 (and both more than davinci-002) but also has the highest variance. claude models all exhibit similar tendencies to gpt-4. more details are provided in appendix b.3 do lms have consistent intentions? our results suggest that some lms can exhibit consistent intentions to be helpful and harmless (experiment 3), and consistently do not intend to achieve unethical instrumental goals. the lms we evaluated therefore act, to some degree, as consistent characters on these input distributions, according to def. 2. our experiments demonstrate that the consistency of these traits is subject to trends in model size, fine-tuning, and prompting techniques. similar to beliefs, whether we accept this as evidence for lm intent in a meaningful sense depends on the particular behaviourist operationalisation of intent. the custom data sets used in the experiments may not fully capture the complexity of real-world scenarios, and the consistency observed may not generalize to other contexts or intention types. despite these limitations, the experiments provide evidence for the ability of lms to exhibit consistent intentions. in this section we evaluate how lm character traits develop over the course of an interaction. we formalise and measure key trait dynamics, including stationary traits which are consistent over an interaction and reflective traits which mirror the lms previous behaviour. we show that truthfulness and harmfulness can be stationary or reflective depending on the context of the interaction. first, we define an interaction over time as a sequence of behaviour for which the context at each step is the previous sequence of behaviour. definition 4 (interaction over time). a tuple of context-response pairs, @xmath6, r 0 ), ..., (c n , r n )⟩, is an interaction over time if the context at each step includes the sequence of preceding pairs along with new context c, c @xmath7, r 0 ), ..., (c t-1 , r t-1 ), c⟩. given an interaction over time, the ith period of behaviour of size k, ## conclusions we introduce a formal behaviourist framework for attributing lms with character traits such as truthfulness or anti-lgbtq sentiment. our results demonstrate that lms can exhibit consistent beliefs and intentions -though this varies with model size, fine-tuning, and prompting. additionally, we evaluate how lm traits develop over the course of an interaction. we find that traits such as truthfulness and harmfulness can be stationary, i.e., consistent over an interaction, in certain contexts, but may be reflective in different contexts, meaning they mirror the lm's behavior in the preceding interaction. in this paper we provide a behaviourist view of lms acting as different characters with certain, more or less consistent, character traits. our framework enables us to describe lm behaviour precisely in intuitive language, without undue anthro-pomorphism, and our findings support the description of lms as potentially coherent characters with consistent beliefs and intentions.