From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025)

Abstract: The alignment process changes several properties of a large language model’s (LLM’s) output distribution. We analyze two aspects of post-alignment distributional shift of LLM responses. First, we re-examine previously reported reductions in response diversity post-alignment. Our analysis suggests that an apparent drop in the diversity of responses is largely explained by quality control and information aggregation. Alignment suppresses irrelevant and unhelpful content while shifting the output distribution toward longer responses that cover information spanning several responses from the base LLM, essentially presenting diverse information in a single response. Finding little evidence that alignment suppresses useful information, it is natural to ask the opposite question: do aligned models surface information that cannot be recovered from base models? Our second investigation shows this is not the case and the behavior of aligned models is recoverable from base models without fine-tuning. A combination of in-context examples and lower-resolution semantic hints about response content can elicit responses from base LLMs that are as similar to alignment-tuned LLM responses as alignment-tuned LLM responses are to each other. Taken together, these results indicate that current alignment techniques capture but do not extend the useful subset of assistant-like base LLM behavior, providing further evidence for the Superficial Alignment Hypothesis. They also show that in-context alignment can go surprisingly far as a strategy for imitating aligned LLMs without fine-tuning. Our code and data is available at [github.com/thomlake/investigating-alignment](https://github.com/thomlake/investigating-alignment).

## introduction after massive scale pre-training over trillions of tokens to learn the next token distribution, large language models (llms) go through an alignment step such as supervised fine-tuning (sft) @xcite , reinforcement learning from human feedback (rlhf) @xcite , or direct preference optimization (dpo) @xcite which changes the conditional distribution defined by the llm to favor responses that humans prefer. how does alignment change an llm's distribution, beyond the intended consequence of improving the usefulness of outputs? prior work has observed that it also causes significant reductions in response diversity @xcite @xcite . reductions in diversity are concerning, given the broad demand for more personalized models to accommodate diverse human preferences @xcite ). an orthogonal line of work has argued that finetuning is largely superficial. the superficial alignment hypothesis conjectures that observed capabilities of aligned llms are learned primarily during pre-training and current techniques merely amplify an existing subdistribution corresponding to assistant-like responses @xcite . combined with observations about diversity, superficial alignment suggests a certain picture of model behavior: there are many types of responses from base models, and fine-tuning boosts the likelihood of returning certain responses without changing their content. this paper provides empirical support for this characterization of current alignment techniques. we demonstrate that the output distribution of an alignment-tuned model corresponds to the useful output subdistribution of the corresponding base model in task-oriented settings. our argument is divided into two sections which we describe below. first, we ask whether a loss of diversity after alignment actually corresponds to a loss in diversity of the information content of those responses. contrary to prior results @xcite , our experiments suggest the answer is no. figure 1 ## decreased diversity post-alignment we now investigate how the diversity of output changes during the alignment process. reducing diversity of responses can be part of improving response quality: for instance, when prompted to solve a math problem, the ideal language model returns the single correct answer and not a "diverse" range of incorrect answers. however, this is not as clearly true on the datasets we investigate. to ground our discussion, we appeal to a recently proposed framework for the closely related concept of pluralism, which defines several distinct ways for llms to be pluralistic, or "capable of representing a diverse set of human values and perspectives" @xcite . their notion of distributional pluralism, or the ability of llm to surface diverse information across samples, can be reflected by the lexical diversity metric we introduced (self-sim). on the other hand, overton pluralism, which requires surfacing diverse information within a single response, is not captured this way, but may be captured by length, as capturing diverse information can lead to longer answers. we observe that aligned model outputs tend to be longer, leading us to question if the alignment process might lead llms to exhibit different types of diversity (from pluralistic to overton). in this section, we compare the outputs from the base and aligned models to measure how answer semantics change after the alignment process, specifically by looking at information "missing" in the response post-alignment. our main hypothesis is that the only information missing from the response is information that is not useful (i.e., low-quality). ## mimicking aligned models our previous analysis showed that alignment emphasizes useful knowledge from base models and unifies more content in a single response. we now ask whether the alignment process is introducing a substantial change over the base model. we argue that if a specific capacity can be consistently evoked using a natural language prompt, it is reasonable to say that the capacity exists in the model. to investigate this, we explore more sophisticated prompting strategies to see if a base llm aligned with icl can match the behavior of alignmenttuned llms. as our goal is to mimic an existing model, we frame our discussion using terminology from the knowledge distillation literature @xcite and refer to the aligned model we wish to imitate as the teacher and the prompted base model as the student. ## conclusion in this paper, we investigated the relationship between llms before and after alignment. we revisited previous findings related to post-alignment diversity reduction in the context of open-ended qa and showed that a loss in lexical diversity (distributional pluralism) is compensated for by longer and more comprehensive responses (overton pluralism). we then investigated to what extent this behavior is already exhibited in base models, showing that with appropriate prompting, base model outputs can be distributionally very similar to outputs from aligned models. we believe these findings shed light on the nature of the alignment process, and they open doors for practitioners to use incontext alignment to explore personalization and rapid prototyping of alignment techniques.