Your Mileage May Vary: How Empathy and Demographics Shape Human Preferences in LLM Responses (2025)

Abstract: As large language models (LLMs) increasingly assist in subjective decision-making (e.g., moral reasoning, advice), it is critical to understand whose preferences they align with—and why. While prior work uses aggregate human judgments, demographic variation and its linguistic drivers remain underexplored. We present a comprehensive analysis of how demographic background and empathy level correlate with preferences for LLM-generated dilemma responses, alongside a systematic study of predictive linguistic features (e.g., agency, emotional tone). Our findings reveal significant demographic divides and identify markers (e.g., power verbs, tentative phrasing) that predict group-level differences. These results underscore the need for demographically informed LLM evaluation.

## introduction large language models (llms) are increasingly used in subjective domains such as moral reasoning and personalized advice-giving @xcite . the growing impact of llms on decisions and communication highlights the need to align them with diverse human preferences. prior work in the nlp community has highlighted variations in human annotations for subjective tasks @xcite @xcite , reflecting underlying differences in individual or group-level preferences. while it was once common to treat a single "gold standard" label as the ultimate target for model alignment, recent work in subjective nlp evaluation increasingly emphasizes demographic representation and diverse perspectives @xcite . among subjective tasks, recent work has evaluated llm responses to emotional or ethical dilemmas using human judgments @xcite @xcite . however, figure 1 : roadmap of our study. we integrate demographic and empathy-level data (yellow) with human ratings of llm responses (green), and connect these ratings to the content and linguistic features of the responses (blue and pink), enabling a multi-dimensional analysis of how user characteristics and response styles influence perceived empathy. in the context of responding to emotional users, the issue of generating empathetic responses has received increased attention (e.g. @xcite and systems displaying empathy have been shown to generally increase user satisfaction (rostami and navabinejad, 2023). however, it is unclear whether users always prefer more empathetic responses across the board or whether some users may prefer other styles. we present a comprehensive analysis of human ratings on llm-generated responses, focusing on the influence of demographic factors and empathy. we examine how different user groups evaluate responses to moral dilemmas, and how these preferences vary across demographic and empathy lines to shape perceptions of llm output. we explicitly distinguish between cognitive and affective empathy in our analysis, which helps reveal how different aspects of empathy relate to variation in response preferences across users @xcite . to the best of our knowledge, we are the first to analyze these questions. to deepen our understanding, we analyze model responses using both quantitative and qualitative methods, leveraging tools such as linguistic inquiry and word count (liwc) and connotation frames of power and agency to identify linguistic features associated with divergent preferences @xcite . finally, we offer actionable insights for adapting llm communication styles to better align with the empathic expectations and demographic characteristics of diverse user populations. for example, we observe that the model most preferred by the participants tends to adopt a more assertive and agentic tone, and groups that have lower empathy tend to favor models that show more prevalence in cognitive linguistic markers. by linking linguistic patterns to demographic preferences and type of empathy in users, this work equips developers to refine inclusivity in llm design and challenges the assumption that "one-size-fits-all" outputs suffice for subjective tasks. our findings highlight the need for human-conscious evaluation in llms, particularly as the field shifts toward subjective notions of quality @xcite . ## related work recently, llms' affective abilities have come to the forefront, evidenced by the emergence of several benchmarks to measure them, such as emobench @xcite and emotion-queen @xcite . in particular, much of this work has focused on generating empathetic responses: a 2024 metareview found that human raters find llm-generated responses to be more empathetic than humans' @xcite . several later studies have compared llm responses to dilemmas to humans', supporting previous findings that llms can respond empathetically @xcite @xcite . although subjective tasks in nlp are a growing area of interest, previous studies on llm responses in this context have not considered individual preferences, despite it being a task with high disagreement @xcite . @xcite collect how empathetic humans are, but only as an attention check, and do not correlate it with response preferences. in this paper, we consider how the human raters' own tendency to empathise, the type of empathy they display, and their sociodemographic features affects response preferences. in terms of preferences, only a few studies have considered what features of a response appeal to human judges and llms @xcite . in the context of eq and empathy, @xcite found that models have distinct response patterns that also differ from humans' by studying the surface features of each model by training a bow classifier, and @xcite investigate different aspects of response quality, such as actionability and sympathy. we present a study aligning preferences with linguistics features. ## experimental setup data we use the dataset developed for the language model council (lmc) framework @xcite , which supports collective evaluation of different llms on a task related to emotion intelligence. this dataset extends the emotionally charged dilemmas emobench benchmark @xcite . specifically, it consists of a total of 100 dilemmas (e.g., a family member asking for money when you are facing financial problems yourself). a council of 20 llms rank each other's open-ended responses to these interpersonal conflicts, resulting in a total of 2,000 evaluated responses. after the generation of llm responses, human participants were asked to evaluate pairs of these responses to assess the models' emotional intelligence. in each of the 1300+ comparisons, participants selected the response they considered the better (e.g., "response a much better than b") and provided qualitative feedback on their choice. this feedback included labels describing emotional qualities, such as "the best response expressed emotions," "the system sympathized with the protagonist," and "the best response seems trustworthy." the full list of survey questions is provided in figure 8 in appendix a. (further details on participants recruitment, quality control are included in the appendix of (zhao et al., 2025)). while zhao et al. (2025) compared llm preferences to human responses using the collected ratings, it did not analyze the underlying human preferences reflected in the qualitative feedback. in this paper, we build on that data to investigate how demographic background and empathy level correlate with preferences for llm-generated dilemma responses. models we use nine models from the llm council for our analysis of human response preferences. these models were selected by @xcite from the original 20 council members for comparison against human evaluations. the models include qwen1.5-110b-chat @xcite , gpt-4o-2024-05-13 (openai, 2024), claude-3-opus @xcite , qwen1.5-32b-chat @xcite , llama-3-70b-chat (platforms, 2024), claude-3-haiku (anthropic, 2024), mixtral-8x7b (jiang et al., 2024), figure 2 : examples of empathy assessment questionnaire sourced from @xcite . the first question assesses affective empathy. the second and third questions assess cognitive empathy. ## human ratings analysis we analyze human evaluators' ratings of llm responses to dilemmas to uncover how preferences vary across demographic groups. here we present the demographic-specific ranking patterns, overall trends in model preferences, and how empathy influences people's preferences. demographic-specific ranking following zhao et al. (2025), we compute demographic-specific model rankings using arena-style pairwise comparisons against a fixed reference model, qwen-1.5-32b. the rationale and procedure for selecting the reference model are detailed in their paper. for each subgroup, we filter the human evaluation data accordingly and calculate rankings based on expected win rates using the elo scoring system @xcite , with bradley-terry (bt) coefficients (bradley and terry, 1952) applied to improve statistical robustness. note that not all subgroups yield a complete set of pairwise results due to limited comparisons within some demographics. we present the key subgroup ranking patterns in figure 3 , and report full rankings for all subgroups in figure 9 in appendix c. ## discussion and conclusion this paper presents an in-depth analysis of how demographic factors and empathy shape human preferences for llm responses. specifically, we investigate how a person's preferences for different llm responses to dilemmas are shaped by their sociodemographic background as well as their degree and type of empathy. we identify those with cognitive empathy and affective empathy depending on their replies to an empathy questionnaire. we then examine each group's preferences for different llms and explore how these preferences align with the linguistic features of the model responses. our results show that both demographic factors and empathy levels shape human preferences in llm responses to emotional dilemmas. for instance, groups with higher affective empathy tend to prefer responses that include emotional validation and compassionate language, whereas those with lower empathy levels prefer more straightforward, practical answers. we also find that language style and emotional tone in llm responses significantly influence these preferences. these findings have several implications for model development and real-world applications: personalization. llms could be tailored to individual user traits, such as cognitive or affective empathy, enabling more satisfying and appropriate responses in emotionally charged contexts like mental health support, coaching, or education. lightweight, opt-in user profiling-via short questionnaires or behavioral cues-could allow llms to dynamically adjust tone and style to align with individual communicative preferences. model selection and fine-tuning. developers can leverage empathy-aligned preferences to guide the choice or fine-tuning of models for specific applications. for example, models exhibiting emotionally validating language may be better suited for grief support or counseling tasks, whereas models with more neutral, pragmatic language may be preferable in task-oriented or efficiency-focused settings. rethinking one-size-fits-all empathy. our findings challenge the default assumption that empathy is always the optimal strategy. in some contexts, neutral or pragmatic responses may better serve user needs. overall, our study highlights the importance of considering both demographic and empathy-related factors in llm design. by accounting for the diverse emotional and communicative needs of users, developers can create systems that are more inclusive, contextually appropriate, and ultimately more effective in supporting human-ai interaction. while empathy has been the de facto strategy used in nlp, several studies have problematised its use from an ethical perspective @xcite . our findings suggest from a usability perspective, there may also been room for more diversity of responses.