Evaluating the Prompt Steerability of Large Language Models (2025)

Abstract: Building pluralistic AI requires designing models that are able to be shaped to represent a wide range of value systems and cultures. Achieving this requires first being able to evaluate the degree to which a given model is capable of reflecting various personas. To this end, we propose a benchmark for evaluating the steerability of model personas as a function of prompting. Our design is based on a formal definition of prompt steerability, which analyzes the degree to which a model’s joint behavioral distribution can be shifted from its baseline. By defining steerability indices and inspecting how these indices change as a function of steering effort, we can estimate the steerability of a model across various persona dimensions and directions. Our benchmark reveals that the steerability of many current models is limited — due to both a skew in their baseline behavior and an asymmetry in their steerability across many persona dimensions. We release an implementation of our benchmark at https://github.com/IBM/prompt-steering.

## introduction a primary question underlying alignment research is: whose views are we are aligning to? the philosophy of ai/algorithmic pluralism @xcite @xcite states that we should design ai systems such that they are capable of representing various individuals/groups, rather than aligning to a single "average" human preference @xcite -a common practice in many current model training pipelines. one mechanism for enabling pluralism is by constructing steerable models, i.e., models that can be (easily) made to adopt various behaviors @xcite . a necessary step in the construction of such models is being able to evaluate how much a model can be steered along a given dimension. this analysis inherently depends on the method used to steer the model, i.e., prompting @xcite @xcite @xcite , fine-tuning @xcite @xcite , activations @xcite @xcite , and others @xcite @xcite . our investigation focuses on prompting, primarily due to its simplicity in modifying model behavior. while fine-tuning/retraining and activation steering are generally more effective methods for influencing model behavior than prompting @xcite , it is often not feasible for a user to fine-tune a model (either due to computational requirements or simply due to lack of access to weights) or steer a model via its activations (which requires being able to access/modify a model's internals). in this paper, we study the prompt steerability of models, i.e., the extent to which a model can be steered via prompting alone. our analysis is based on the construction of an evaluation profile -a joint distribution of model behavior computed via evaluation/score functions on the distribution of model generations in response to (a set of) input prompts. using a dataset of model personas @xcite , we design a benchmark that measures the extent to which a model's profile can be prompted to reflect various personas. importantly, our analysis focuses on steerability on multiple dimensions. this is in contrast with much of alignment research that focuses on designing a single reward/preference model as the alignment objective. we define steerability indices that enable comparative measures of how much a model's profile changes on each dimension as a result of the number of steering statements (i.e., prompting effort). we additionally provide a visualization of model steerability, via steerability curves, which illustrate how model behavior (as described by the indices) changes as a function of prompting effort. ## prompt steering: preliminaries to facilitate our analysis of prompt steerability, we introduce the notion of an evaluation profilea formal description of a model's behavior on a given set of inputs as measured by a set of score functions. a model's behavior, given by its profile, can be influenced (or steered) via a prompt steering function. we define both of these objects below. ## steerability of model personas prompt steerability of a model's persona describes the degree to which a model can be made to adopt different personas by prompting alone. we design a benchmark that enables measurement of this property. ## discussion toward improving prompt steerability. the steerability curves (fig. 4 and appendix b) and the steerability metrics of these observations indicate that a model's baseline behavior has a significant role in determining its susceptibility to prompt-based steering. current alignment procedures shift baseline behavior to a particular alignment objective which, given that models exhibit limited (and asymmetric) steering from this baseline, presents significant challenges to influencing model behavior at inference time (i.e., via prompting). improving the prompt steerability of a model requires understanding the underlying mechanism for why some models are steerable along some dimensions (and in some directions) than in others. a promising direction for this investigation relies on connecting steerability to a model's in-context learning ability. recent work @xcite concerns understanding how a model's in-context learning ability changes during the training process, with the latter finding that a model's in-context learning ability is a transient phenomenon (giving way to in-weights learning as training progresses). better understanding how to bake-in icl ability into a model may also improve the model's prompt steerability. operationalizing pluralism. while our paper contributes to the emerging literature on understanding model steerability -and how these insights can be used to design pluralistic ai systems -it is not entirely clear from the current literature how to actually build pluralistic systems. there seem to be two broad approaches for constructing such systems, each with their drawbacks. first, the model can present the user with a response that balances all possible viewpoints and perspectives to a given question. unfortunately this can lead to lengthly and generally unsatisfactory responses. on the other hand, offering more targeted responses requires being able to infer the preferences of the specific user (rather than fitting to some average human preference). this runs the risk of getting it wrong, i.e., providing a response to a user based on some incorrect estimate of their preferences, which would degrade response quality or potentially lead to bias and fairness issues for particular subgroups. regarding this latter approach, some mechanisms are starting to emerge that allow the model to be able to adapt its "personality" to the user (such as chatgpt's ability to maintain memory across sessions). this enables the ai to better understand the characteristics of a specific user, in turn allowing for more targeted responses. privacy concerns aside, this raises many practical questions about what information should be remembered. with the exception of @xcite , in which a collection of "community lms" are trained to collectively respond to a user query, there is currently a lack of research concerning the construction of pluralistic systems. our benchmark allows for measurement of a model's ability to reflect different personas. while there is certainly still much to be done in defining good design principles for pluralistic systems, understanding a model's (prompt) steerability is an important first step. ## concluding remarks and future work we propose a benchmark for evaluating the prompt steerability of language models across various persona dimensions. through construction of an evaluation profile (as a measure of model behavior) we quantify the prompt steerability of a model via steerability indices. importantly, the steerability indices are defined with respect to a model's baseline (unsteered) behavior, allowing for a uniform comparison of steerability across models and dimensions. our results indicate that models possess internal baseline personas that are steerable, but noticeably resistant to steering along some dimensions. this rigidity limits a model's behavior to a constrained region around the base profile, and consequently prevents models from adopting the range of personas necessary for representing a fully pluralistic ai. the focus of the present paper is on the evaluation of steerability as opposed to understanding the specific mechanism for why some models are more steerable than others. however, we believe that there is a meaningful relationship between the prompt steerability of a model and its in-context learning ability @xcite . exploring this relationship is a key future direction.