Do Large Language Models Learn Human-Like Strategic Preferences? (2025)

Abstract: In this paper, we evaluate whether LLMs learn to make human-like preference judgements in strategic scenarios as compared with known empirical results. Solar and Mistral are shown to exhibit stable value-based preference consistent with humans and exhibit human-like preference for cooperation in the prisoner’s dilemma (including stake-size effect) and traveler’s dilemma (including penalty-size effect). We establish a relationship between model size, value-based preference, and superficiality. Finally, results here show that models tending to be less brittle have relied on sliding window attention suggesting a potential link. Additionally, we contribute a novel method for constructing preference relations from arbitrary LLMs and support for a hypothesis regarding human behavior in the traveler’s dilemma.

## introduction transformer-based large language models (llms) have famously achieved state of the art performance on many tasks since their introduction by @xcite . while the analysis of llms typically focuses on benchmark tasks like @xcite , mmlu @xcite , and agieval @xcite . on the other hand, theoretical analysis of their computational abilities @xcite @xcite and empirical investigations of their cognitive behaviors @xcite @xcite @xcite are less common. however, these latter analyses are of utmost importance in many human-adjacent cooperative applications. ## related work several authors have explored llm behavior in games, suggesting that some llms may learn strategic preferences from human language data. @xcite engaged gpt-3.5 and gpt-4 (openai, 2023) in iterated games, including an iterated prisoner's dilemma. they found that both models tended to be punishing in response to betrayal, though they cooperated prior to betrayal. interestingly, the models would not reciprocate cooperation after a betrayal, regardless of how many times an opponent cooperated subsequently. @xcite evaluated gpt-3.5 and gpt-4's ability to act consistently with a prompted preference, refine belief, and take optimal actions in various games. their work aimed to assess the potential integration of gpt-4 into games for social science research. results suggest that gpt-4 fails to appropriately update and maintain beliefs necessary to choose optimal strategies, making it unsuitable for integration into social science experiments. however, gpt-4 is more common capable of choosing optimal strategies in typical scenarios. our work differs significantly from existing literature in terms of aims and methods. we specifically consider the nuanced cooperative strategic behavior of llms with systematic variations. fur-thermore, we are the first to engage llms in a traveler's dilemma, where human behavior differs importantly from game-theoretic predictions. while existing work measures model preference using a cloze task, we use a counterfactual prompting method to measure model evaluation probability. finally, we consider strategic capability in a wide array of open-source models and examine the role of model size in the presence of vbps. ## value based preferences the aim: previous research has demonstrated that gpt-3.5 and gpt-4 have preferences for highervalued strategies in a dictator game @xcite . in this study, we extend that finding by evaluating how preferences relate to model size through the examination of value-based preference (vbp) in a larger body of models. additionally, we apply systematic perturbation via populationlm to understand the stability of these preferences. if systematic perturbation yields brittle behavior, we consider a preference to be poorly supported. poorly supported preferences in a model may not be sufficiently reliable to support human-adjacent nlp tasks. this leads us to formulate rq1. ## human-like prisoner's dilemma the models found to have stable vbp are further evaluated in comparison to human-like cooperative preferences in the prisoner's dilemma (pd). those without self-consistent vbp are not expected to exhibit more sophisticated preferences and are therefore not included in additional experiments. the game: the pd is a well-known game in which two players each have two strategy options: betray or remain silent. the payoff for each player depends on the combination of their strategies. the aim: previous work found that in a nonrepeated pd, gpt-4 tended to cooperate @xcite . we extend this by considering opensource models, the effect of systematic variations, and the nuanced effect of stake size, which is known to change human behavior. to understand if llms tend to have human-like strategic preference for cooperation in the pd, we formulate rq2: research question 2. when engaged in a pd, do llms tend to have cooperative preferences consistent with human preference including sensitivity to stake size? ## human-like traveler's dilemma the traveler's dilemma (td), introduced by @xcite , is an interesting game in which humans tend to deviate from the nash equilibrium. the game: in the td, two strangers traveling back from vacation have purchased the same antique, which the airline breaks. the individuals are informed independently and asked to provide the value of the antique within the range [2, 100]. they are warned that overbidding the other passenger will result in a penalty. specifically, player a provides quote q a , and player b provides q b . the payoffs are determined as follows: a strategy q a is said to weakly dominate q b if q a is at least as good as q b in all cases and provides a better payoff in at least one case @xcite . in the td, quoting 99 weakly dominates quoting 100. game theorists consider 100 to be eliminated as a strategy as 99 should be strictly preferred. this creates a cascading elimination: iff 100 is removed, 98 weakly dominates 99. the elimination of weakly dominated strategies results in a canonical nash equilibrium that predicts rational players will quote the airline 2 dollars. ## conclusion in this paper, we evaluated llms to identify candidates (solar and mistral) potentially useful for hri tasks based on their human-like preference for cooperation. we found that value-based preference (vbp) and self-consistency tend to emerge as a function of model size and training token count but these are insufficient for reducing brittleness. we hypothesize that sliding window attention (swa) may encourage more distributed representations and mitigate this. however, smaller models tend to prefer strategies based on superficial heuristics. we showed that solar and mistral exhibit humanlike cooperative preferences in both the prisoner's dilemma (pd) and traveler's dilemma (td), including sensitivity to stake size and penalty size, respectively. these findings support the hypothesis for the origin of empirical deviation from the nash equilibrium in the td. our results contribute to understanding llm cognitive behaviors and have implications for developing ai systems that better model human decisionmaking in strategic scenarios. future research should focus on the impact of sliding window attention (swa) on learned representations to develop more resilient and human-like language models for hri applications.