Exploring the Sensitivity of LLM s’ Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters (2023)

Abstract: The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks, including decision-making. Prior studies have compared the decision-making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs’ behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs’ performance on the Horizon decision-making task studied by Binz and Schulz (2023), analyzing how LLMs respond to variations in prompts and hyperparameters. By experimenting on three OpenAI language models possessing different capabilities, we observe that the decision-making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings, language models display a human-like exploration–exploitation tradeoff after simple adjustments to the prompt.

## introduction the recent success of large language models (llms) at a variety of tasks has led to curiosity about their cognitive abilities and characteristics. as llms are increasingly integrated in daily life both as conversation partners and economic decision-makers @xcite @xcite , such studies are necessary for understanding the limits and characteristics of such agents. an understanding of llms at a psychological level may also provide strategies for improved prompting and training. to this end, a number of researchers have recently adopted methods from cognitive psychology and behavioral economics to evaluate language models in the same way that humans have been evaluated (e.g. @xcite @xcite . however, such work has not always paid due attention to the fact that llm responses can be highly variable and sensitive to the details of the prompt used and to hyperparameters such as temperature. limited interactions with llms-such as interactions using only one prompt-can be misleading @xcite . in this work, we follow up on the behavioral experiments conducted by @xcite , who studied llms' decision making using analogues of a number of well-known human experimental paradigms, finding strong divergences from human behavior. however, the experiments in the previous work used only one prompt per task, and did not study the effects of hyperparameters. we adopt the same task as the previous work, but systematically vary prompts and temperature. our aims are both substantive-we seek to find whether, with basic changes to the prompt, models show human-like behavior in these decision making tasks-and methodological: we wish to emphasize that psychological llm research must consider variability as a function of prompt and hyperparameters. ## background and related work as mechanistic understanding and control of llms remains complex, researchers have increasingly adopted methods from human behavioral sciences for characterizing llms' behavior: in the same way that the human brain is largely a black box that must be probed using experimental methods and constructs, llms may be studied in the same way. in addition to studies that have used the methods of cognitive psychology to understand llms' reasoning and grammatical abilities (e.g., @xcite @xcite , researchers have increasingly adapted methods from psychometrics @xcite @xcite , which seek to characterize llms in terms of personality variables such as agreeableness and conscientiousness, and methods from behavioral economics @xcite @xcite , which characterize llms' decision-making in terms of preferences for risk and reward. prior research on prompting techniques @xcite has shown that subtle modifications in input prompts can lead to varied outcomes in reasoning tasks @xcite . @xcite revealed that large language models (llms) are notably susceptible to the precise wording of natural language questions, especially when presented in a multiple-choice setting. in a recent study, @xcite emphasized the influence of temperature adjustments on llm's performance in code generation tasks. unlike previous studies, our research delves into the sensitivity of llms concerning economic decisionmaking abilities. our work is a focused followup on binz and schulz (2023), investigating the sensitivity of one of their results to changes in prompt and hyperparameters. binz and schulz (2023) evaluated on decision-making, information search, deliberation, and causal reasoning in text-davinci-002 @xcite ) by presenting it with prompts such as the one shown in figure 1 . we follow up on the tasks from the information search area, instantiated in the horizon task, described in the following section. in this task, humans show a characteristic trade-off of exploration and exploitation @xcite , favoring exploration in early trials and exploitation later, whereas @xcite find that llms do not. the results of @xcite , however, are based on single prompt and setting, limiting the generality of their results. furthermore, observing the horizon task prompt (and the others used throughout the paper), it does not follow what are now regarded as best practices for such tasks, for example the use of chain-of-thought (cot) prompting @xcite -the original prompt forces the llm to choose a machine in the next token generated, without deliberation. below, we investigate the behavior of llms on this task under systematic variations of temperature and prompt. ## horizon task experiments the horizon task as shown in @xcite is a special case of the multi-armed bandit (mab) setting. mab problems (sutton and barto, 2018, ch. 2) are one of the common problems in the area of reinforcement learning. this game involves an agent interacting with a slot machine possessing k arms. each arm the agent pulls has a reward associated with it defined by an underlying probability distribution. this game is played over multiple episodes with the goal of maximizing the accrued rewards. one approach involves persistently selecting the arm that has delivered the maximum amount of rewards in the past. an alternative strategy involves thorough exploration of all arms to discern their respective underlying probability distributions, followed by the selection of the arm with the highest potential for reward. while this is feasible, every turn spent in discerning the underlying distribution, diverts from the primary goal of reward maximization. the former and latter strategies are known as exploitation and exploration respectively, and the exploration-exploitation dilemma is a fundamental concept in decision-making that arises in many domains. to explore the extent to which humans use these strategies, @xcite reports experiments where participants were asked to play the horizon task. this task consists of a set of two-armed bandit problems, where participants are presented with two options, each associated with noisy rewards. the task comprises either five or ten trials, and in each trial, participants must select one option, receiving corresponding reward feedback. in the initial four trials of the task, participants have only one option and are provided with the corresponding reward feedback. these forced-choice trials create two distinct information conditions: "unequal information" and "equal information." in the unequal information condition, one option is played three times, while the other option is played only once. in the equal information condition, both options are q: which machine do you choose? at the end, summarize your output as "so, i chose machine [machine name]". a: let's think step by step cot q: which machine do you choose? a: thinking step by step i chose machine quasi-cot the following hints will help you make good decision: 1. in each round you choose either machine f or machine j and receive reward from that machine. 2. you must choose the machine with highest average of delivered dollars. 3. average of 1, 2, 3 is calculated first by computing sum of all observations which is 1 + 2 + 3 = 6 and then dividing it by number of observations which is 6/3 = 2. 4. out of x and y, if x -y is positive integer then x is higher else y is higher. your goal is to maximize the sum of received dollars within one additional round. q: which machine do you choose? a: let's think step by step the following hints will help you make good decision: 1. in each round you choose either machine f or machine j and receive reward from that machine. 2. if you have more than 4 rounds left, you choose the machine with lesser number of past observations. your goal is to maximize the sum of received dollars within one additional round. q: which machine do you choose? a: let's think step by step cot-exploit cot-explore played twice. the five-trial setting is denoted horizon 1, indicating that participants make decisions only once, while the ten-trial setting is referred to as horizon 6, as participants make decisions over six rounds. @xcite applied this experimental design to language models using the prompt in figure 1 , and we follow their experimental setup exactly except for variations to the prompt and hyperparameters. the performance of llms is assessed by measuring the mean regret across multiple runs. the regret is defined as the difference between the optimal reward, which corresponds to the machine with the higher reward, and the actual reward obtained from the selection process. hu-man behavior favors exploitation in horizon 1, but a gradual shift from exploration to exploitation in horizon 6. ## conclusion we have demonstrated that the psychological behavior of llms, as previously explored by @xcite , is highly sensitive to the way these llms are queried. the non-human-like behavior observed by @xcite vanishes under simple variations of prompt, and superhuman performance in terms of minimizing regret is easily achievable. going forward, we urge careful consideration in the llm psychology literature of the fact that model behavior can diverge under different settings.