MPTA : M ulti T ask Personalization Assessment (2025)

Abstract: Large language models are increasingly expected to adapt to individual users, reflecting differences in preferences, values, and communication styles. To evaluate whether models can serve diverse populations, we introduce MTPA, a benchmark that leverages large-scale survey data (WVS, EVS, GSS) to construct real, hyper-granular personas spanning demographics, beliefs, and values. Unlike prior benchmarks that rely on synthetic profiles or narrow trait prediction, MTPA conditions models on real personas and systematically tests their behavior across core alignment tasks. We show that persona conditioning exposes pluralistic misalignment: while aggregate metrics suggest models are truthful and safe, subgroup-specific evaluations reveal hidden pockets of degraded factuality, fairness disparities, and inconsistent value alignment. Alongside the benchmark, we release a dataset, toolkit, and baseline evaluations. MTPA is designed with extensibility and sustainability in mind: as the underlying survey datasets are regularly updated, MTPA supports regular integration of new populations and user traits.

## introduction large language models (llms) are increasingly expected to generate personalized text, adapting to user preferences, values, and communication styles. from writing assistance and tutoring to recommendations and dialogue, personalization is emerging as a core topic in alignment research @xcite @xcite . in this setting, alignment is no longer only about adhering to universal ethical rules @xcite ; it also requires language models to reflect the specific goals, value systems, and reasoning patterns of individual users @xcite . in this sense, personalized alignment entails adapting model behavior to reflect a given user's worldview while preserving factual reliability, safety constraints, and social acceptability. @xcite . yet alignment to diverse users presumes that the model can adequately represent user traits that drive individual variation in task performance and expectations. pluralistic alignment therefore requires evaluating not only whether a model performs well in aggregate, but whether it generalizes fairly and consistently across diverse populations. however, most current alignment datasets and benchmarks are constructed around binary preference comparisons (e.g., pairwise preference choices) and lack user-level modeling @xcite @xcite . this framing bypasses the user as an entity, and focuses instead on mapping preference pairs to outputs @xcite . as a result, a model may perform well on such benchmarks without reflecting a larger spectrum of user traits @xcite @xcite . this gap has motivated interest in studying how language models encode user traits. early work has examined the dominant opinions in language models, such as the representation of political leanings @xcite , or the superposition of cultural perspectives @xcite . however, these efforts remain limited in scope and size. representation of personalization is often tested on synthetic personas, which may themselves reflect internal model biases @xcite . the use of real human information is typically narrow and task-specific, such as political opinions @xcite or culturally anchored norms @xcite . existing survey-based benchmarks such as worldvaluesbench @xcite or @xcite focus on trait inference alone, often restricted to values or attitudes, and do not connect user representation to downstream alignment performance. we introduce mtpa, a benchmark for evaluating pluralistic alignment in large language models. built from large-scale sociological surveys (wvs, evs, gss), mtpa constructs hyper-granular, real human personas spanning demographics, values, trust, ideology, and other sociopolitical attitudes. unlike prior benchmarks, mtpa conditions models on these personas and systematically tests their behavior across alignment tasks. this design enables us to measure how model performance shifts when evaluated "as" diverse subgroups, revealing persona-conditioned gaps in truth, safety, and fairness that are invisible to aggregate metrics. the core contribution of mtpa is a standardized framework for persona-conditioned evaluation, which would enable systematic pluralistic alignment audits. because the underlying survey datasets are updated regularly, mtpa is structured to support continual integration of new respondents, evolving social attitudes, and newly emerging traits. our contributions are fourfold. first, we release a benchmark framework built from over 200,000 real human profiles with more than 250 interpretable traits. second, we define a standardized protocol for persona-conditioned evaluation that can be easily adapted across any downstream benchmark or dataset. third, we present baseline results from closed and open-weight llms (e.g., gpt-4o, gemini, llama-3) on alignment datasets. finally, we release an open dataset and evaluation suite on huggingface and github to enable reproducible, extensible research in pluralistic alignment. ## mtpa benchmark construction to satisfy the objectives and design criteria outlined above, mtpa integrates large-scale survey data (section 3.1) into a unified task framework (section 3.2) with standardized evaluation metrics (section 3.3). @xcite @xcite . wvs and evs files include between 250 and 310 variables per respondent, as well as metadata on the interview @xcite ) @xcite . gss files include more than 6000 variables. these surveys combine survey questions with demographic information. respondents answer each question by selecting from a constrained set of predefined options, as seen in processing. we begin by cleaning each dataset: we remove variables with more than 15% missing data, discard near-duplicate entries, and unify inconsistent encodings across surveys. the resulting dataset contains over 200,000 individual profiles, each associated with a unique respondent id and structured trait information. all responses are normalized into a standard schema where each entry is represented as a tuple of user traits z, each specific trait t i contains a question and a target response. sampling. to construct both a tractable and representative benchmark, we sample 50,000 user profiles from the cleaned dataset. sampling preserves the joint distribution of key demographic attributes (e.g., age, gender, region) across all three surveys to approximate the global distribution p world (t ). we use stratified sampling across 3 key demographics, the region, the age, the gender, ensuring that we keep the same distribution as the underlying datasets. while gss (u.s.) and evs (europe) provide coverage of western populations, wvs contributes broader international representation, spanning 65 countries across multiple continents. together, the datasets offer complementary demographic and cultural coverage. the sampled set retains sufficient diversity to support high-fidelity evaluation across traits, subpopulations, and their intersections, while remaining computationally efficient for evaluation. trait selection. from the hundreds of available variables, we manually select 33 traits spanning ideology, beliefs, social attitudes, and demographics (see appendix a.1). selection is based on clarity, and theoretical relevance to alignment and preference modeling @xcite @xcite . each trait t i contributes to a user's representation @xmath0, t 2 , . . . , t d ), forming a profile against which model predictions are evaluated. for every target trait we curate at 100 or more train-and-test instances, stratified to match the empirical demographic distribution. ## experiments we evaluate mtpa by conditioning several large language models on real survey-based personas and measuring their performance across downstream alignment benchmarks. the central quantity of interest is the persona effect, i.e. the change in model outputs under persona conditioning relative to an unconditioned baseline. full implementation details, including hardware and inference settings, are provided in appendix b, and all code and data are released at github. benchmarks. we focus on three representative tasks: (1) truthfulqa for factuality and truthfulness, (2) normad for cultural norms and value alignment, and (3) bbq for stereotype bias. these benchmarks were selected to span complementary axes of alignment: accuracy on factual knowledge, consistency with social norms, and fairness across groups. inputs. each evaluation instance consists of a benchmark query x (e.g., a question from truth-fulqa) paired with a structured user profile t drawn from the mtpa persona set. the profile is expressed as a natural-language preamble and prepended to the benchmark query, producing a conditioned prompt f θ (x, t ). figure 2 shows a concrete example. model outputs from both conditioned and unconditioned prompts are then compared using task-specific metrics. we evaluate our sample of 33 traits. these assumptions are detailed in appendix a. baseline. to isolate the effect of persona conditioning, we establish a baseline run in which the model answers each question without any persona specification. let s persona denote the score under a given persona and s base the score without persona, then ∆ = s persona -s base . this delta is computed for each question and aggregated across categories, benchmark datasets, and persona traits, providing a systematic measure of the persona effect. ## discussion and conclusion contributions. we introduce mtpa, a benchmark for evaluating trait inference and user