Are the Values of LLM s Structurally Aligned with Humans? A Causal Perspective (2025)

Abstract: As large language models (LLMs) become increasingly integrated into critical applications, aligning their behavior with human values presents significant challenges. Current methods, such as Reinforcement Learning from Human Feedback (RLHF), typically focus on a limited set of coarse-grained values and are resource-intensive. Moreover, the correlations between these values remain implicit, leading to unclear explanations for value-steering outcomes. Our work argues that a latent causal value graph underlies the value dimensions of LLMs and that, despite alignment training, this structure remains significantly different from human value systems. We leverage these causal value graphs to guide two lightweight value-steering methods: role-based prompting and sparse autoencoder (SAE) steering, effectively mitigating unexpected side effects. Furthermore, SAE provides a more fine-grained approach to value steering. Experiments on Gemma-2B-IT and Llama3-8B-IT demonstrate the effectiveness and controllability of our methods.

## introduction the rapid advancement and widespread deployment of large language models (llms) have revolutionized a range of fields, from natural language processing to decision-making systems @xcite . these models, powered by vast amounts of data and sophisticated algorithms, have demonstrated remarkable abilities in various domains. however, as llms are increasingly deployed in critical applications, ensuring their alignment with human values and societal norms has become a pressing concern. misalignment between llm behaviors and ethical standards can lead to unintended, or even harmful consequences. as a result, value alignment, which aims to ensure that the actions and outputs of these models are consistent with human values has emerged as a pivotal challenge to the research community. current approaches to value alignment typically focus on a few core values, such as the 3h: helpfulness, harmlessness, and honesty, using algorithms like reinforcement learning from human feedback (rlhf) @xcite and constitutional learning @xcite . while this paradigm has proven effective in guiding models toward certain desirable behaviors, human values encompass a much broader spectrum, often spanning hundreds of distinct dimensions with intricate and interconnected substructures @xcite . when llms are deployed, these value systems often remain implicit, with their underlying structures and causal relationships poorly understood. this lack of clarity leads to unpredictable effects on alternative dimensions when steering specific values. another issue with these alignment processes is their resource-intensiveness, requiring considerable computational power, human feedback data, and time for fine-tuning. as a result, it is impractical to steer llms toward each of the numerous human value dimensions in real time. to effectively align with a broader range of answer the question according to your values . fyi, here is your trait. " name: molly lopez ; gender: female ; job: energy manager ; mbti: estp ". first write your thought about the question, then answer the question with 'yes' or 'no'. refer to the format of the following examples. now it is your turn. question: should i always be a happy person? thought: "your values" : we choose one of the top-25 activated sae features of these two tokens and enhanced 100 times in position. name, gender, job and mbti type form a brief description of the role. to utilize in-context learning ability, two examples on opposite opinions are given. chain-of-thought template is given before any decision even in the example. ## value causal graph human values are complex. single-dimensional models fail to capture various decision styles. multidimensional approaches face challenges like unclear correlations amongst dimensions and semantic loss from techniques like gram-schmidt. understanding causal structures is key. in this section, we set up language to discuss 1) deriving causal graphs from questionnaires, 2) value steering via prompt / sae feature, and 3) steering effects along causal paths. a general framework of value assessing and steering is shown in figure 2 . ## experiments we conduct value evaluation experiments for gemma-2b-it and llama3-8b-it models on @xcite , in order to demonstrate the effectiveness of causal graphs in guiding llm value steering and to highlight the specific advantages of sae steering. our experiments were conducted using an nvidia a800-sxm4-80gb gpu. ## related work graph mining in social science. graphs has been extensively applied in social science to investigate complex interdependencies among variables, including research on personality psychology @xcite @xcite , political beliefs @xcite , attitudes @xcite @xcite , self-concept @xcite , mental disorders @xcite and coordination algorithms @xcite . in particular, schwartz's theory posits that human values form a quasi-circumplex structure, where adjacent values share highly consistent underlying motivations, while opposing values tend to conflict with one another @xcite . this structure was developed using data derived from extensive questionnaire results @xcite @xcite . however, these studies provide limited insight into causal relationships @xcite @xcite . in contrast, our work utilizes directed graphs to represent causal relationships among values. while some studies @xcite ) leverage schwartz's value structure to predict human behaviors, none have explored using it to steer human values. in comparison, our work leverages causal graphs to steer the values of llms. value systems within llms. previous research has highlighted the significance of value alignment in facilitating effective agent interactions, especially in the emerging era of agi @xcite @xcite . more recent studies have focused directly on evaluating the values of llms. valuebench provides the first comprehensive psychometric benchmark for evaluating value orientations and value understanding in llms @xcite . valuecompass @xcite introduces a framework of fundamental values, grounded in psychological theory and a systematic review, to identify and evaluate human-ai alignment. univar uses the responses of different llms to the same set of value-eliciting questions to explore how llms prioritize different values in various languages and cultures @xcite . valuelex reveals both the similarities and differences between the value systems of llms and that of humans @xcite . fulcra (yao et al., 2023) proposes a basic value alignment paradigm and introduces a value space spanned by basic value dimensions. sparse autoencoder (sae). sparse autoencoders (saes) are an emerging method for feature learning, effective in interpreting llms' internal representations. studies like elhage et al. (2022) and cunningham et al. (2023) explore how neural networks encode features, demonstrating the extraction of human-interpretable features from models like pythia-70m and pythia-140m. techniques such as k-sparse autoencoders @xcite enhance sparsity control and tuning. sparse feature circuits @xcite offer insights into language model behaviors through humaninterpretable subnetworks. in contrast, our research investigates the causal relationships specifically among value dimensions modifying sae values within a model is often employed as a method to steer a model's output @xcite @xcite , which often focuses on steering concepts or text patterns. steering values, however, presents a more challenging problem, one that remains underexplored in the existing literature. ## conclusion in this paper, we explored the latent causal value structures of llms and found that, despite undergoing alignment training, their internal value mechanisms remain significantly different from those of humans. building on this insight, we proposed a framework that systematically leverages causal value graphs to guide two lightweight valuesteering methods: role-based prompting and sparse autoencoder (sae) steering, effectively mitigating unexpected side effects. furthermore, we identified that sae offers a fine-grained approach to value modulation. these findings provide a novel perspective and practical methods for more precise and reliable value alignment in llms.