Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback (2025)

Abstract: As language models have a greater impact on society, it is important to ensure they are aligned to a diverse range of perspectives and are able to reflect nuance in human values. However, the most popular training paradigms for modern language models often assume there is one optimal answer for every query, leading to generic responses and poor alignment. In this work, we aim to enhance pluralistic alignment of language models in a low-resource setting with two methods: pluralistic decoding and model steering. We empirically demonstrate that model steering offers consistent improvement over zero-shot and few-shot baselines with only 50 annotated samples. Our proposed methods decrease false positives in several high-stakes tasks such as hate speech detection and misinformation detection, and improves the distributional alignment to human values from different demographics. We hope our work highlights the importance of diversity and how language models can be adapted to consider nuanced perspectives.

## introduction recent advancements in natural language processing, driven by reinforcement learning from human feedback (rlhf), have garnered a significant amount of interest @xcite . experts have begun adopting large language models (llms) in applications with increasing social impact. with this rapid adoption, there are many emerging challenges with ai alignment, ensuring that automated systems are developed in the best interest of humans. the very definition of "best interest" is nuanced and open to exploration. in both reinforcement learning and fine-tuning settings, traditional machine learning conventions assume that there is one optimal answer @xcite . in cases of extreme diverging preferences, this causes a reward model to prefer generic responses that do not fully satisfy anyone @xcite . guiding language models to reflect multiple perspectives, also known as pluralistic alignment @xcite , is essential for high-stakes tasks in law, medicine, and finance that are often dependent on diverse preferences. ai alignment is an open-ended research question with significant room for exploration, and interdisciplinary collaboration is essential for ensuring proper representation of a task's diversity @xcite . one advantage to llms is the ability to improve the system with in-context instructions, which allows for more semantically rich feedback from domain experts and facilitates stronger collaboration. it is also important to consider the cost of alignment -training strategies are resourceintensive in both compute and annotations, which is often expensive for domain-specific tasks. in practice, a domain expert also cannot provide feedback for every sample at inference time. we wish to explore a sparse setting with minimal data, where a domain expert can quickly specify their perspective in a few samples. our main contributions are as follows: • we propose to adapt language models to diverse preferences in a low-resource (i.e. sparse) setting using model steering with sparse auto-encoders (saes). this pipeline adds flexibility for altering model behaviour without training, reducing the requirements for training data and allowing quicker adaptation to novel tasks. ## alignment to sparse feedback our full pipeline is shown in figure 1 . we incorporate diverse expert feedback from multiple perspectives using a variation of contrastive decoding, which we call pluralistic decoding. this can easily be applied to any black-box language model with exposed logits. however, changing the input query is often a limited model intervention. in preliminary experiments we found pluralistic decoding does not significantly change the model output. with access to the intermediate model residuals, we additionally explore model steering to predefined perspectives using sparse auto-encoder (sae) representations. preliminaries. an llm refers to a causal language model that samples some next token prediction t ∼ p(x) over the token space for an input @xmath0, x 2 , . . . , x n } of length n. as shown in equation ( 1 ), an llm can be thought of as a composite function of l transformer decoder blocks, where the representation of the input f l (x) at layer @xmath1. defining feedback. for an arbitrary perspective, represented as an annotator a, alignment is defined as conditioning the logit space on some natural language feedback c a , i.e. p(x|a) ≡ p(x|c a ). with llms, experts can improve performance with natural language feedback, which allows for more semantically rich feedback from domain experts. we distinguish between coarse and granular feedback. coarse feedback refers to top-down guidelines for alignment (e.g."detect harmful statements") while granular feedback is specific to a particular sample (e.g."this is targeting a population of people"). granular feedback is easier to quantify, as it is more concrete than values or principles @xcite . however, it is often more expensive to obtain this feedback, especially in high-stakes domains such as law, medicine, and finance @xcite . we wish to explore data-sparse methods where a domain expert can quickly specify their perspective to maximize their valuable insights. if the sae is able to capture a well-formed representation, then it would create some consistent alignment to an expert at inference. for a validation set of n samples, we take the sae vector s a as shown in equation (2). in practice, we encode each contrastive pair into the encoding space, taking the difference with and without feedback in the input, and averaging the differences over n samples. pluralistic decoding. once we have multiple responses to the query from various perspectives, next is the non-trivial task of combining them into one answer. we adapt contrastive decoding @xcite to multiple generations, which we call pluralistic decoding (pd). as defined in equation ( 3 ), the final distribution p(x|a) for a set of annotators @xmath2. intuitively, this encourages a higher weighting for distributions with more uncertainty, or more plausible token predictions. when not using pd, we perform a simple mean of the logits, i.e. p(x|a) = 1 |a| @xmath3. ## experimental settings datasets implementation details can be found in appendix c. we use the following datasets: ## results and discussion baseline results. our main results are shown in alignment scales with data sparsity. for the full feedback setting, we use all available feedback in the gold labels, or synthetically generated with the same process as previous works @xcite , and combine the predictions into one distribution with pluralistic decoding (pd). this process gives a consistent performance boost over the zero-shot baseline, especially for the gqa dataset which decreases 0.10 points in js distance for llama3.1-8b. we believe this is because the gqa has the most dense feedback, since we are using synthetically generated feedback compared to manual annotations from lawyers. on gemma2-9b, the results are even stronger with full feedback. while the few-shot performance indicates the possibility of lexical artifacts in the lhs dataset, we observe pd improves performance beyond few-shot prompting for gemma2-9b. with gemma2-9b on mislc, full feedback enhances performance on the positive class by 6.3 points f1 score. with the exception of gemma2-9b performance on lhs, we observe relatively consistent improvements over the zero-shot baseline. this indicates there is some merit to implementing model steering for alignment, although full feedback is still preferable when possible. increased alignment for domain-specific tasks with sparse feedback. in applications such as content moderation, experts are more concerned with understanding the potential utility of a system rather than its absolute performance @xcite , so precision is often a more important metric than f1 score. sae model steering decreases the number of false positives in the llama3.1-8b experiments on both legal datasets without sacrificing accuracy on the positive class. however, the performance on the positive class is still low, indicating there is no change in the task understanding. we also do not observe improvements when combining model steering and pluralistic decoding in an end-to-end pipeline. we theorize this is because we are intervening on the transformer residual, we are straying too far from the learned space. for example, ppo places a kl divergence penalty in its training loss to prevent the model from straying too far from the pre-trained information. since model steering is simple addition in the sae vector space, we do not place such constraints. previous works @xcite demonstrate increasing the magnitude of the steering vector also results in nonsensical outputs, indicating there are limits to the degree of intervention. layer choice in sae vector steering. we investigate the choice of sae layer in figure 2 . while there are some general trends in performance, there are also fluctuations and nonlinear trends that are likely affected by the underlying information stored at each layer of the transformer. these are also the n s t h r n w h r a b t o s r e d t o s y t c c _ 3 1 8 0 200 400 600 label no yes unsure setting value figure 5: the label distribution of each class for steering on the lhs dataset for llama3.1-8b. nst refers to no steering, i.e. the zero-shot setting. alignment to diverse feedback. we decompose the mislc and lhs task to examine label predictions per steering vector at the chosen layer, as shown in figure 4 and figure 5 . the first two annotators give similar predictions but the third has more predictions to the positive class (yes). this results in a higher recall but lower precision, which balances out to a similar bin-f1 score. in the lhs task, there are three distinct groupings of hate speech definitions -human rights laws (hr), social media policies (tos), and criminal offences (cc). if our method is sufficiently aligned to the feedback, then the criminal code (cc) would predict the positive class the least frequently. we do find that one vector aligned to a social media policy (tosyt) gives a higher positive rate, but the other vectors are at a similar level. one interesting observation is the negative class (no) is predicted the most on cc_318, corresponding to the most severe definition of hate speech in the lhs dataset (advocating genocide). this demonstrates there is some alignment to the label distribution of the coarse feedback used to generate the steering vector. we believe the steering vectors improve alignment to the stricter legal definitions compared to the base model. please refer to appendix c for comparison of coarse-grained and fine-grained feedback on one dataset. alignment alignment is the general field of study towards ensuring ai aligns with human values @xcite . it is an active area of research with many open questions: first, it is somewhat difficult to know how to define all possible dimensions of human values @xcite . for example, an llm agent learns to lie or deceive the user because it was never rewarded for honesty @xcite . steering model steering is an emerging field of study on directing language model behaviour by its own internal representations, rather than updating parameters through training @xcite . sparse autoencoders (saes) are a promising direction for model steering. previous works demonstrate saes produce more interpretable features that detangle polysemantic representations @xcite , and improve the ability to steer granular behaviours @xcite . there are works that have successfully applied steering to control knowledge selection @xcite or helpfulness @xcite , but none have tried steering over multiple dimensions. please refer to appendix a for more related work. ## conclusion in this work, we explore how to increase ai alignment to a diverse range of human values. we introduce pluralistic decoding and utilize model steering via sparse auto-encoders to increase the signal of human feedback in low-resource settings. we empirically demonstrate that model steering offers consistent improvement over the baseline for 50 annotated samples, and demonstrate a decrease in false positives on domain-specific tasks. however, this method does not enhance the task understanding or underlying model reasoning. we hope this work highlights the importance of pluralistic alignment and inspires future work in the area.