One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity (2025)

Abstract: Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models’ internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM “populations” by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models’ value alignment and decreasing the diversity of their conceptual representations.

## introduction as large language models (llms) have become more sophisticated, there has been growing interest in using them to replace human labor. this appeal of llms has even made its way to settings where human behavior itself is the object of inquiry: recently, researchers have proposed that llm-generated responses can be used in place of human data for tasks such as polling, user studies, and behavioral experiments (e.g., @xcite @xcite @xcite . if possible, using a synthetic replacement for the process of human data collection could be transformative for a variety of human factors disciplines, from political science to economics and psychology. in both domains, there is both individual-and population-level variation. it is possible that individual variation overlaps with the population average (homogeneous population) or separates from it (heterogeneous population). our experiments are designed to tease these two options apart. while prior work suggests that llms can capture certain behavioral patterns, there are ongoing debates as to whether they are valid replacements for human subjects @xcite @xcite . one key issue is whether llms capture conceptual diversity: the variation among individuals' representations of a particular domain. a natural way to study conceptual diversity is to study the variability in llms' response distributions at the population level -that is, by considering averages across simulated individuals. however, population-level variability can be a flawed measure of conceptual diversity for several reasons. first, it assumes certain charac-teristics about the nature of people's mental representations that may not hold in practice -for example, that people have high certainty in these representations (e.g., @xcite , or that people's responses are a deterministic best-guess, with no additional information contained in successive guesses (e.g., @xcite . second, assessing population-level variation without individual-level variation can mask important information about the population, particularly whether it is homogeneous (comprised of individuals who share similar underlying representations) or heterogeneous (comprised of individuals with meaningfully different representations) (see figure 1 ). conceptual diversity may be negatively impacted by alignment, which further complicates the picture. post-training alignment techniques, such as rlhf @xcite and rlaif @xcite , are now standard parts of llm development, and presumed to contribute to the human-like abilities of models @xcite . however, it has also been observed that "aligned" models show biases that can limit the lexical and content diversity of their outputs @xcite @xcite . it also remains unknown whether alignment to synthetic preferences (instead of human preferences) might worsen these biases, as it is possible these models "collapse" when recursively trained on synthetically generated data @xcite . in this paper, we (1) investigate the ability of llms to capture human-like conceptual diversity, and ( 2 ) analyze the effect of alignment by comparing conceptual diversity across non-aligned models to models aligned using rlhf or rlaif. in our experiments, we first simulate populations of unique individuals in llms using two techniques proposed by prior literature: temperature-and promptbased manipulations. we then evaluate ten opensource llms on two domains with rich human behavioral data: word-color associations, and conceptual similarity judgments. while there is no single agreed-upon metric for capturing conceptual diversity, we consider two different metrics that are each well-suited to their respective domains, to measure the conceptual diversity of syntheticallygenerated llm "populations". we find that no model approaches human-like conceptual diversity. further, aligned models generally display less conceptual diversity than their non-aligned, fine-tuned counterparts. our findings suggest that there may be trade-offs between increasing model safety in terms of value alignment, and decreasing other notions of safety, such as the diversity of thought and opinion that models represent. we caution that these trade-offs should be better understood before models are used as replacements for human subjects, or deployed in human-centered downstream applications. ## background recent works have proposed using llms as standins for humans in many applications and domains, including opinion surveys and polling in political science @xcite , user studies in humancomputer interaction @xcite , annotation tasks @xcite , and various economic, psycholinguistic, and social psychology experiments @xcite . several works have responded by identifying llms' lack of answer diversity as a potential harm of using llms to replace human subjects. these works have focused on whether models are capable of capturing demographic-and subgroup-level variation in settings where accurately simulating the social identities of a human population of interest is directly relevant to the task. @xcite find that inference-time interventions to improve models' output diversity do not prevent llms from misportraying and flattening the representations of already marginalized demographic groups. even after explicit steering toward certain demographics, llms still do not capture these group's responses on public opinion surveys @xcite . these observations have motivated recent attempts to go from broad, population-level observations of diversity, to metrics that assess finergrained notions. for example, he-yueya et al. (2024) measured the extent to which lms capture human knowledge distributions on real-word domains like language learning and mathematics. their metric captures whether two populations are similarly sensitive to the relative difficulty of test items. however, this metric is not well-suited for domains that lack ground-truth answers or where people's mental representations are not easily parameterized. @xcite propose another item-level account of diversity, from the perspective of bayesian statistical modeling. they find that llms do not capture human variance at the item-level in an experiment on pragmatic language, and aggregate item-level predictions to replicate the basic logic of our experiments is as follows. first, we identify domains where prior work has demonstrated rich conceptual diversity in humans. second, we test whether llms capture conceptual diversity patterns by simulating "populations", using temperature-and prompting-based manipulations. finally, we compare the conceptual diversity achieved by models against that of humans, and specifically examine the effect of alignment on the degree of fit between models and humans. ## discussion in both domains, the base llama and tulu models generated a high number of invalid responses (text generations that did not mention a valid color/target word). we hypothesize that for llama, as the only model in our suite that had not been instruction fine-tuned, this indicates a fundamental difficulty comprehending the task instructions. however, for tulu, (which had been instruction fine-tuned, but not aligned) this could indicate weaker understanding of subjective, non-fact-based domains like the ones we test. further, our findings that models aligned through both rlhf and rlaif display less conceptual diversity suggests that the preference information captured by pairwise judgements in such feedback datasets could be too narrowly constraining the post-training latent space of models. exploring alternative paradigms of value alignment that allow for more complete pictures of human values (cf. zhi-xuan et al.) and further studying models' internals pre-and post-alignment could help overcome such issues. our findings also corroborate those of recent work in many domains that suggest that injecting variability in models' outputs through temperature and prompt manipulations does not do enough to induce meaningful variability on relevant metrics, and often just leads to highly incoherent outputs (invalid responses). one reason for this could be that increasing the entropy of the output distribution or conditioning on highly surprising tokens will increase models' uncertainty over subsequent tokens in a domain-general sense. however, the kind of uncertainty that gives rise to meaningful individual differences in humans is likely much more constrained to a particular task and cognitive domain than this. exploring more task-specific methods for injecting structured randomness into models' generations could help llms to overcome their population-average behavior in ways that more meaningfully simulate the cognitive differences in human individuals. this raises further questions of what factors give rise to individual differences in humans and whether these are the same attributes that can result in individual-like models. while it is common to obtain representative samples of a human population for studies in cognitive science using identity attributes like gender or race, the identity-level representations of people that are most easily replicated in post-training methods likely do not capture the reasons for the differences in people's conceptual representations. exploring corresponding notions of context-specific cognitive or computational resources (e.g., working memory) in models could allow them to serve as testbeds for theories of individual differences in humans (cf. @xcite . ## conclusion here, we used a new approach to evaluate conceptual diversity in synthetically-generated llm "populations". specifically, we adopted metrics from prior studies of conceptual diversity in humans that relate the variability in individuals' underlying representations to population-level variability. we evaluate two popular methods for eliciting unique individuals from llms -temperature and prompting manipulations -in the word-color and wordconcept domains and find that adding noise in the form of unrelated and randomly shuffled prompts does more to increase model diversity than persona prompting and temperature manipulations. across two domains, no model reached human-level conceptual diversity. further, our results suggest that alignment to synthetic or human preferences (in the form of rlaif and rlhf) flattens models' conceptual diversity in these domains, compared to their base or instruction fine-tuned counterparts. while alignment has now become a central part of llm development, our findings urge caution that more work should be done to understand how it affects models' internal diversity of "opinions" or conceptual representations. this is especially important given recent calls to use llms as stand-ins for human subjects in behavioral research. while it is clear that there could be gains under this paradigm, we should also ask: what might we be losing?