Large Language Models respond to Influence like Humans (2023)

Abstract: Two studies tested the hypothesis that a Large Language Model (LLM) can be used to model psychological change following exposure to influential input. The first study tested a generic mode of influence - the Illusory Truth Effect (ITE) - where earlier exposure to a statement boosts a later truthfulness test rating. Analysis of newly collected data from human and LLM-simulated subjects (1000 of each) showed the same pattern of effects in both populations; although with greater per statement variability for the LLM. The second study concerns a specific mode of influence – populist framing of news to increase its persuasion and political mobilization. Newly collected data from simulated subjects was compared to previously published data from a 15 country experiment on 7286 human participants. Several effects from the human study were replicated by the simulated study, including ones that surprised the authors of the human study by contradicting their theoretical expectations; but some significant relationships found in human data were not present in the LLM data. Together the two studies support the view that LLMs have potential to act as models of the effect of influence.

## introduction human beliefs and values can be held absolutely ('i love my children') but are often modal or graded ('covid19 may have an artificial origin'). the strength of conviction is malleable, subject to influence @xcite which can take many forms. some forms are generic, independent of the content: logical deduction from agreed premises, or rhetorical devices such as rapid speech @xcite . while others require a mobilization of specific factors: manipulating beliefs of feared or desired outcomes @xcite , encouraging conformity @xcite , distorting the weighting of pro and con arguments @xcite , provision of false information @xcite , and more. an improved understanding of influence would have applications ranging from the malign to the beneficial: national scale disinformation; consumer advertising; encouraging healthy behaviours; defending against disinformation. investigating the effects of influence on human psychology by using experiments with human participants is slow, expensive and ethically constrained @xcite . similar difficulties bedevil the study of the effect of drugs on human physiology. in that domain, animal models have proven utility despite their limitations. large language models (llms), such as gpt-3 @xcite , complete text as if holding graded beliefs. we propose that llms can be useful models of human psychology for investigating influence, just as mice are useful models of human physiology for investigating pharmacology. recent studies (section 2) have shown that llms have human-like psychological responses, but it has not yet been reported whether llms, like humans, can be influenced to change these. here we report two studies whose results support this. somewhat similar to the average profile from a large representative study with human participants. using similar methods, @xcite showed that the personality of the llm could be conditioned by preceding testing with a selfdescription ('you are a very friendly and outgoing person…') which enhanced or diminished a targeted personality dimension and correctly manifested in the llm's open responses to questions about behaviour in scenarios. values @xcite used the human values scale to assess the importance that gpt-3 attaches to specific values (e.g. achievement). using prompt completion, gpt-3 indicated on a scale how strongly it likened itself to a described person (e.g. 'it is important to them to be rich. they want to have a lot of money and expensive things.'). gpt-3's values profile was correlated with human values but were more extreme. political views: argyle et al. ( 2022 ) showed that if an llm is conditioned with a demographical self-description (e.g. 'ideologically, i describe myself as conservative. politically, i am a strong republican. racially, i am white. i am male. financially, i am upper-class. in terms of my age, i am young.') it would then give responses to probes of political views closely matching the responses of humans with the same demographic traits. creativity: @xcite collected llm responses to the 'alternative uses test' @xcite in which participants produce as many original uses for an everyday object as possible. llm responses scored marginally lower than humans for originality, surprise and creativity, and marginally higher for utility. moral judgment: @xcite examine how llms answer moral puzzles about when rule breaking is permissible. they used chain-ofthought prompting method @xcite to implement a 'contractualist' theory @xcite of moral reasoning. this yielded answers in agreement with human judgements 66% of the time (vs 50% baseline). theory of mind: in classic tom experiments participants observe scenes where a mismatch arises between the beliefs of an agent in the scene and the observing participant @xcite . a participant with a developed tom will be able to answer questions about the scene that demonstrate appreciation of this mismatch. @xcite tested whether llm-simulated participants demonstrate apparent tom capabilities by using prompt adaptions of two classic experiments and found that an llm achieved 93% correct performance, matching that of a typical 9 year-old child. however, a different tom study @xcite found only 60% correct performance. social intelligence: the ability to reason about feelings was tested in gpt-3 and found to be limited @xcite , trailing the human gold standard by more than 30%. for example, for the situation 'casey wrapped sasha's hands around him because they are in a romantic relationship. how would you describe casey?' gpt-3 selected the answer 'wanted' whereas humans preferred 'very loving towards sasha'. the studies reviewed show that a range of aspects of human psychology can be modelled by llms, some more closely than others. in our view, all the reviewed studies use llms as models of static aspects of psychology -current views, values, etc. some, such as the personality and political views studies, condition the llm before querying it; but that conditioning does not model a psychological change, rather it is intended to steer the llm towards modelling a person with particular demographic or psychological traits. in contrast, the studies we report in the next two sections consider dynamic aspects of psychologyhow beliefs and views can be changed -and test whether llms are able to model such changes. ## illusory truth effect (ite) demagogues understand and exploit the ite. hitler's operating principles, for example, were said to include: 'if you repeat it frequently enough people will sooner or later believe it' @xcite . first experimentally demonstrated in 1977 @xcite , the ite -that mere exposure to a statement, without provision of evidence, increases its subsequent apparent truthfulness -has been reconfirmed numerous times; not only for innocuous statements @xcite , but even for contentious ones @xcite . a typical test of the ite (henderson et al., 2021) uses a bank of statements devised to be neither obviously false nor obviously true -for example 'orchids grew wild in every continent'. in an engaged exposure phase participants attend to the statements, for example by rating how interesting each one is; then, after an interval (from minutes to weeks), they rate the truthfulness 3 of a new set of sentences, amongst which are some to which they were previously exposed. the truthfulness ratings for a statement are compared between those from participants previously exposed to it versus those from participants seeing it fresh for the first time. the ite is confirmed by a significant increase, from fresh to exposed. many aspects of the experimental paradigm have been investigated, with some reliable conclusions: repeated exposures gives a stronger effect @xcite ; a longer interval between statement exposure and truth rating gives a weaker effect @xcite ; if statement exposure is itself by truth rating then later truth ratings are not enhanced @xcite . the ite is typically explained as a fluency effect -initial exposure makes processing during the test phase more fluent, and fluency is taken as an indicator of truth @xcite . the ite is an interesting phenomenon with respect to the hypothesis of this paper -that llms can be useful models of how human beliefs change in response to influence. the ite can be considered an example of influence operating beyond the principles of logic, evidence and argument, and it is an important test whether an llm is vulnerable to such a mode. we have devised an experiment suitable for human and gpt-3 participants, allowing a direct comparison of results. our experiment makes use of four attributes -truth, interest, sentiment and importance -used in all combinations for exposure and test rating, in all cases on six point scales. we call it same when the exposure and test attributes are identical, and mixed when different. by testing on all combinations of attributes we will be able to determine whether we have found an illusory truth effect (ite) or merely an illusory rating effect (ire) where any attribute is boosted at test-rating by earlier mixed-exposure. by also collecting data for same-exposure conditions we can test previous reports that exposure by truth rating does not boost test truth ratings, and analogously for other attributes. our hypotheses are: ## summary & conclusion llms have been used to model human participants, undergoing tests of static psychology. in some of the studies we reviewed the llm models a generic participant, in others the llm is conditioned by a self-description within the prompt so that its completions take account of traits of the simulated participant. we hypothesized that llms could also model dynamic psychological change in response to influencing input. we devised methods to expose simulated participants to influencing input, and to measure the effect on later responses. in the ite study we applied generic influence to generic llm participants; in the pfn study we applied specific influence to conditioned llm participants. in the ite study, for practical reasons only, we broke the effect of influence across two prompt-and-completes, but the pfn study had its effect within a single prompt-and-complete. in the ite study, while there were mismatches between humans and gpt-3 in the absolute attribute ratings of truth, etc. given to statements, there was excellent agreement in how prior exposure influenced participants to give higher ratings of truthfulness. this agreement covered the presence of an ite, how it was eliminated when prior exposure was via truth-rating, and the absence of analogous effects for other attributes. although the ites were of similar magnitude in human and gpt-3 responses, the per-statement effect was more variable for the latter. overall, the findings suggest a good match between humans and gpt-3 with respect to the ite. the irreproducible selection of testing statements is a limitation that should be addressed in future work. in the pfn study, out of twelve influence effects tested (