Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations (2025)

Abstract: Large-scale surveys are essential tools for informing social science research and policy, but running surveys is costly and time-intensive. If we could accurately simulate group-level survey results, this would therefore be very valuable to social science research. Prior work has explored the use of large language models (LLMs) for simulating human behaviors, mostly through prompting. In this paper, we are the first to specialize LLMs for the task of simulating survey response distributions. As a testbed, we use country-level results from two global cultural surveys. We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions for a given question. Then, we show that this method substantially outperforms other methods and zero-shot classifiers, even on unseen questions, countries, and a completely unseen survey. While even our best models struggle with the task, especially on unseen questions, our results demonstrate the benefits of specialization for simulation, which may accelerate progress towards sufficiently accurate simulation in the future.

## introduction humans are diverse, and they hold diverse opinions. this is why surveys are essential tools for informing decision-making in policy and industry as well as social science research. running large-scale surveys, however, is often costly and time-intensive. large language models (llms) have demonstrated promising potential for simulating human behaviors across groups and individuals @xcite @xcite . llm simulations of survey responses, if accurate towards the corresponding populations, could accelerate social science research and aid in more informed policy decisions. out of the box, however, llms are known to generate erroneous, stereotypical, or overconfident answers, especially in culturally diverse contexts @xcite , which limits their usefulness for survey simulations. prior work has at most tried to improve simulation accuracy through prompting strategies @xcite @xcite . in this study, our goal is instead to specialize llms for survey simulation and gain a better understanding of how good llms can be at simulating survey responses when trained to do so, rather than how good they are when prompted out of the box. as our main testbed, we use countrylevel response distributions from the widely-used world values survey (wvs; @xcite . when prompted with a survey question (e.g., "in your opinion, should the use of nuclear power in japan be reduced, maintained at its current level, or increased?"), corresponding answering options (e.g. "reduced", "maintained at current level", "increased") and a target country (e.g. "japan"), we want our model to predict the distribution over the answering options for that target country. to spe-cialize llms for this task, we devise a fine-tuning method based on first-token probabilities, where the goal is to minimize divergence between predicted and actual country-level response distributions for a given survey question. as shown in figure 1 , we train models on one set of questions and countries from the wvs, then evaluate on both seen and unseen countries and questions as well as another completely unseen survey. across seven llms from three model families, our fine-tuning method substantially boosts prediction accuracy on seen and unseen wvs countries and questions. these results also hold for a completely unseen survey, i.e. the pew global attitudes survey. simultaneously, we find the performance of even the best-fine-tuned models to be far from perfect, especially on unseen questions. we also find that all llms we tested, whether fine-tuned or not, are less diverse in their predictions across countries than the actual human survey data. in summary, we make following three main contributions: other work has explored prompting strategies and frameworks for improving simulation accuracy @xcite @xcite . relatedly, several works have used survey questionnaires, including the wvs used in this paper, with the goal of evaluating values and opinions reflected in llms rather than simulating human survey responses @xcite @xcite @xcite . in contrast to these efforts, our focus is on specializing llms to simulate group-level survey response distribution, which could aid in survey data collection. while we do measure the performance of llms out of the box (zero-shot), our main goal is to investigate the extent to which we can improve llm performance on simulating group-level survey response distributions through fine-tuning, and explore their potential as specialized tools for social science research. ## methodology to address the challenges of simulating culturally diverse survey responses, we introduce a framework for first-token alignment fine-tuning for distribution prediction, designed to improve generalization across populations and survey questions. ## results we investigate two primary research questions: rq1 how effectively does the proposed alignment method improve the distribution simulation of the model on unseen countries and questions? rq2 what is necessary to perform well on the task-how much is dependent on modeling the prior distribution, and how much on context sensitivity? we present comprehensive experimental results to address the two research questions. ## conclusion in this paper, we explored the task of specialising llms to simulate survey response distributions across diverse countries and questions. for this task, we devised a fine-tuning method based on first-token probabilities. our experiments demonstrate that fine-tuning models substantially improves response simulation prediction compared to zero-shot models, for both seen and unseen countries and questions. further, fine-tuned models also show improved generalization to an entirely new survey dataset. despite these improvements, our results also highlight systematic limitations of the models, particularly when simulating responses to unseen questions. we also observed that the models, whether fine-tuned or not, were less diverse in their predictions compared to the human survey response data, raising questions about their utility. while our results provide clear evidence for the benefits of specializing llms for survey simulation tasks, they also underscore the need for caution when using llms for this task, as even the best-performing models exhibited systematic inaccuracies, especially in culturally diverse contexts.