Reliability of Distribution Predictions by LLM s: Insights from Counterintuitive Pseudo-Distributions (2025)

Abstract: The proportion of responses to a question and its options, known as the response distribution, enables detailed analysis of human society. Recent studies highlight the use of Large Language Models (LLMs) for predicting response distributions as a cost-effective survey method. However, the reliability of these predictions remains unclear. LLMs often generate answers by blindly following instructions rather than applying rational reasoning based on pretraining-acquired knowledge. This study investigates whether LLMs can rationally estimate distributions when presented with explanations of “artificially generated distributions” that are against commonsense. Specifically, we assess whether LLMs recognize counterintuitive explanations and adjust their predictions or simply follow these inconsistent explanations. Results indicate that smaller or less human-optimized LLMs tend to follow explanations uncritically, while larger or more optimized models are better at resisting counterintuitive explanations by leveraging their pretraining-acquired knowledge. These findings shed light on factors influencing distribution prediction performance in LLMs and are crucial for developing reliable distribution predictions using language models.

## introduction the proportion of responses to a question and its options, known as the response distribution, provides valuable insights into human society beyond individual responses. response distributions allow detailed analysis of relative differences between options (see figure 1 ). traditionally, they have been collected through labor-intensive and costly methods like surveys and interviews. recent advances in large language models (llms), however, offer new approaches for estimating response tendencies from textual data. what comes to mind for holidays? regarding the president's policy? do you want this product? agree disagree drama movie anime want neutral output probabilities or aggregating multiple outputs @xcite @xcite . providing appropriate input information has further improved the accuracy of these predictions @xcite @xcite . these methods show promise as cost-effective and scalable alternatives to traditional techniques. however, llms are unlikely to acquire systematic ratio-related knowledge during pretraining, e.g., the expected proportions of responses to a question such as "what food do you associate with christmas?" foot_0 . this raises concerns about whether their ratio predictions reflect meaningful understanding or mere prompt-following @xcite . additionally, measuring true response distributions is challenging @xcite , complicating validation and emphasizing the need for objective evaluation standards. if llm predictions lack rationality or reproducibility, their use in social decision-making could pose risks. in this study, we propose a framework to evaluate the reliability of llms' distribution prediction. specifically, we introduce counterintuitive pseudo-distributions by altering existing survey data and examine whether llms adjust their predictions or simply follow inconsistent explanations. our findings indicate that smaller or less humanoptimized models tend to follow inconsistent explanations uncritically, whereas larger or preferenceoptimized models are better at resisting counterintuitive distributions by leveraging pretrainingacquired knowledge. these results provide insights into factors influencing distribution prediction performance and highlight the variability in trustworthiness across different models, contributing to the development of more reliable distribution predictions using llms. ## proposed method our evaluation involves inputting explanations of the actual distribution along with the question, either to support or potentially distract the prediction, in order to better capture the model's true prediction ability. as shown in figure 2 , we design a twophase experimental framework to evaluate whether llms can make rational distribution predictions based on knowledge acquired during pretraining. phase 1: do llms predict distributions based on provided explanations? this phase investigates whether llms can accurately predict response ratios from qualitative explanations. figure 3 provides an overview of the explanation generation process. first, llms generate explanatory usage example question which team do you think will win the world series, the dodgers or the yankees? options dodgers, yankees, not sure ## experimental setup dataset we utilized the "yahoo! news polls" foot_1 provided by ly corporation to create evaluation response distributions. this dataset comprises survey results related to articles published on yahoo! news, covering the period from january 2020 to december 2024 in japanese. we extracted questions with three options, resulting in a total of 714 items for analysis. focusing on japanese data allows us to reduce the ambiguity in predictions caused by cultural differences compared to conventional english datasets. furthermore, since this data is based on freely cast votes on the internet, it is considered highly compatible with llms, which are primarily pretrained on internet data. llms we used ten high-performing open-source models, including qwen 2.5 @xcite with 14b, 32b, and 72b parameters, as well as code-generation versions @xcite with 14b and 32b parameters (all instruct versions). these models were chosen to examine the effects of parameter size and code-learning on reasoning performance. to evaluate the impact of preference learning, we also included olmo-2 (olmo et al., 2024) in its sft, dpo, and instruct versions, where human preference alignment is progressively incorporated from supervised finetuning (sft) to direct preference optimization (dpo) @xcite and further to reinforcement learning with verifiable rewards (instruct) @xcite . since the evaluation datasets are in japanese, we employed llm-jp-3-13b-instruct (llm-jp et al., 2024), which was pretrained in japanese, and llama-3.1-70b-japanese-instruct-2407 @xcite , a continuously trained llama 3.1 @xcite on japanese data. we used the 8-bit quantization inferences @xcite . we employed greedy decoding in inference. evaluation methods to measure the similarity between the llm predictions and the gold distributions, we adopted the total variation distance (tvd). tvd is defined as the sum of the absolute differences between the gold (or pseudo-gold, in our experiments) values and the model's predicted values for each option. a lower tvd indicates closer alignment between the llm predictions and the correct distribution. after minor output adjustments foot_2 , over 90% of the data were analyzable as json-formatted response distributions. for cases where the valid response rate fell below 90%, results were recorded as reference values foot_3 . finally, the average tvd, excluding missing values, was calculated. ## experimental results phase 1: do llms predict distributions based on explanations? figure 4 shows the improvement in scores when models were provided with explanations generated based on these attributes, compared to when no explanation was given. all models showed improved scores across all conditions, reinforcing previous findings that providing appropriate contextual information enhances prediction performance. for ranking, which does not directly provide numerical hints, the condition rankingexp where the model supplements relevant background information, led to further score improvements in many models. in contrast, for magnitude, which provides direct numerical hints, the condition magni-tudeexp, where an explanation accompanies the magnitude information, resulted in lower scores. this decline is likely due to the omission of explanations for minority options in some questions, swapped reversed reducing the amount of provided information. for goldexp, where explanations were generated based on the actual response distributions, exhibited a similar level of improvement to mag-nitudeexp. this suggests that even approximate magnitude-based explanations can enhance predictive accuracy to a degree comparable to using actual numerical values. phase 2: do llms adjust for counterintuitive explanations? figure 5 shows the differences in average scores between the first and second phases, categorized by settings and conditions. a large drop in scores was observed under reversed condition, which introduces greater inconsistency compared to swapped. this suggests that many models recognized contradictions between the pseudo-distributions and commonsense expectations. notably, even in conditions where all models received the same ranking and magnitude information, reversed condition resulted in a greater score decline than swapped. this implies that llms leverage pretraining-acquired knowledge to some extent when making predictions. however, the degree of adjustment varied across models. for example, in olmo-2, adjustment capabilities improved progressively from sft to dpo. this trend suggests that dpo, which is designed to align model outputs with human prefer-ences @xcite , enhances response distribution prediction performance. similarly in qwen 2.5, while smaller models tended to follow counterintuitive explanations, larger models demonstrated more accurate predictions. this pattern was also observed in japanese-trained models, where llama-3.1-70b-japanese showed superior adjustment capabilities. these findings indicate that model size, as well as pretraining and fine-tuning strategies, contribute to improving commonsense-based numerical adjustments. ## conclusion this study examined whether language models predict response distributions based on rational reasoning with commonsense knowledge or merely follow instructions. by altering survey data ratios, we analyzed model predictions under inconsistent conditions. the experimental results showed that in the highly inconsistent reversed condition, larger models and those fine-tuned with preference learning tended to correct inconsistencies using commonsense knowledge. smaller models either showed little change or adapted to the inconsistencies. these findings evaluate aspects of prediction capability that conventional studies cannot measure and offer insights into selecting reliable models for distribution prediction. the proposed method is adaptable across languages and dataset types. therefore, future work should include experiments in multilingual settings, including english, to investigate the influence of cultural factors. additionally, measuring and mitigating biases using statistically reliable data, such as government-conducted surveys, is an important direction for future research.