Are Large Language Models ( LLM s) Good Social Predictors? (2024)

Abstract: With the recent advancement of Large Language Models (LLMs), efforts have been made to leverage LLMs in crucial social science study methods, including predicting human features of social life such as presidential voting. Existing works suggest that LLMs are capable of generating human-like responses. Nevertheless, it is unclear how well LLMs work and where the plausible predictions derive from. This paper critically examines the performance of LLMs as social predictors, pointing out the source of correct predictions and limitations. Based on the notion of mutability that classifies social features, we design three realistic settings and a novel social prediction task, where the LLMs make predictions with input features of the same mutability and accessibility with the response feature. We find that the promising performance achieved by previous studies is because of input shortcut features to the response, which are hard to capture in reality; the performance degrades dramatically to near-random after removing the shortcuts. With the comprehensive investigations on various LLMs, we reveal that LLMs struggle to work as expected on social prediction when given ordinarily available input features without shortcuts. We further investigate possible reasons for this phenomenon and suggest potential ways to enhance LLMs for social prediction.

## introduction social prediction is one of the crucial elements in social studies @xcite , with a body of literature @xcite @xcite devoted to estimating inaccessible features, either unobserved or missing, based on observed ones. historically, social prediction is made by statistical models such as linear regression @xcite . with the development of machine learning, supervised methods have been adopted, e.g. random forest and neural networks @xcite . however, the classic machine learning methods notably rely on extensive labeled training data, which is labor-intensive, especially in social studies. additionally, the predictive power of machine learning * these authors contributed equally to this work. methods is limited @xcite and can hardly model the complicated phenomenon in social life. with the rapid advancement in large language models (llms), undertaking text-related tasks is empowered with a new paradigm @xcite @xcite @xcite . the extensive world knowledge @xcite and inference abilities @xcite enable llms to mitigate the limitations of classic machine learning methods in social prediction. recent works leverage llms in predicting or simulating human responses, such as voting decisions @xcite and political attitude @xcite . they take advantage of llms to augment existing datasets with previously inaccessible features due to unobservability, data missing, sensitivity and privacy issues. promising performance is reported. however, the common methodology of these works is worth being skeptical about: it first creates datasets with a well-constructed survey; next, except for the response feature, any other features are candidates as input, even if they and the response feature are almost semantically equivalent and thus nearly (in)accessible. this methodology introduces a question: if the observed features are nearly equivalent and thus nearly accessible, why did the original survey avoid directly collecting the key response feature, yet bother to predict by other features? this issue hinders the exploration of llms' authentic social prediction ability and the underlying mechanisms, as well as the realistic and practical implementation of proposed methods. our paper responds to it, critically checking and revising social prediction in a group of settings considering the accessibility of features. to study it, our preliminary investigation revisits the famous case of voting prediction @xcite with llms. we define shortcut as the observed features approximately equivalent and (in)accessible with the response feature, which should be masked in input. the result indicates that llms' performance is bolstered by the presence of shortcuts to the desired response features. specifically, the presence of shortcuts directly associated with the feature to be predicted leads to exceptional performance, even replacing llms with machine learning models. unfortunately, this effectiveness comes with a decline when eliminating shortcuts (detailed in sec.2 and sec.3). this performance gap leads us to question the true capability of llms in social predictions, challenging the prevailing perception of their prowess @xcite . the research community is urged to be more cautious and skeptical when employing this method. furthermore, we shed light on the potential causes and solutions to the under-performance, hinting at future works to comply with the realistic settings when delving into social prediction studies. our contributions are listed below: ## revisit voting prediction with llms large language models (llms) have demonstrated impressive performance in predicting voting decisions in the united states @xcite . in this section, we revisit this voting prediction study with llms @xcite and take a further step beyond it. ## social profile prediction in this section, we introduce a social prediction task evaluating llms' predictive power without shortcuts. first, we coin a social prediction dataset based on survey data and methods to eliminate shortcut features. then we introduce three settings to simulate real-world scenarios. finally, we demonstrate and discuss the performance of llms' prediction in new settings. ## discussions some may wonder if the degraded results are caused by suboptimal or even trivial prompts: are there other prompts that can make good predictions? we admit there is a possibility, but this goes beyond the range of our paper. the prompts can be augmented by better-crafted prompts or examples of labeled data (so-called few-shot), but the settings will be incomparable with the prior works, and also ## conclusion in this study, we introduce a survey-based social prediction task to assess the llms' predictive ability using general features. through the replication of experiments and ablation studies of voting prediction tasks, we reveal a significant performance gap between input prompts with and without shortcut features. to further study the llms' predictive ability, we propose a real-world survey dataset with rigorously selected features. based on it, we demonstrate the inability of llms to predict social features only with general features. furthermore, our empirical studies further showcase the potential reasons that constrain the llms' predictive power. in our future research, we aim to explore the efficient methods of providing supervision signals and reference information to improve llms prediction performance. moreover, with the abundant social survey and online data, we plan to use fine-tuning methods to fit the llms knowledge with social prediction tasks.