Large Language Models are biased to overestimate profoundness (2023)

Abstract: Recent advancements in natural language processing by large language models (LLMs), such as GPT-4, have been suggested to approach Artificial General Intelligence. And yet, it is still under dispute whether LLMs possess similar reasoning abilities to humans. This study evaluates GPT-4 and various other LLMs in judging the profoundness of mundane, motivational, and pseudo-profound statements. We found a significant statement-to-statement correlation between the LLMs and humans, irrespective of the type of statements and the prompting technique used. However, LLMs systematically overestimate the profoundness of nonsensical statements, with the exception of Tk-instruct, which uniquely underestimates the profoundness of statements. Only few-shot learning prompts, as opposed to chain-of-thought prompting, draw LLMs ratings closer to humans. Furthermore, this work provides insights into the potential biases induced by Reinforcement Learning from Human Feedback (RLHF), inducing an increase in the bias to overestimate the profoundness of statements.

## introduction gpt-4 has now achieved the ability to perform a wide range of tasks on par with humans @xcite . given that current llms excel at text interpretation and generation, recent work has assessed llms' ability to perform tasks beyond simple language understanding. and the evidence for llms' ability to perform these tasks is changing rapidly: while previous versions of llms (e.g., gpt-3.5) fell short in tasks seen as unsolvable with textual input alone @xcite , more recent models, such as gpt-4, have been suggested to achieve them @xcite . however, most of these studies have focused on presenting the llm with statements following conversational maxims. thus, when presenting prompts to an llm, one generally strives to make the statement informative and truthful to convey one's ideas or goals successfully. however, not all language use is aimed at efficiently communicating information. sometimes, it can be beneficial for a speaker to obscure the meaning of an utterance, for example, to deceive or persuade or to hide one's true intentions. successful, human-level communication requires the listener to detect such language use; otherwise, they are susceptible to deception. here, we assess whether gpt-4 and other llms can identify language created with the aim of impressing the listener rather than communicating meaning. the term "pseudo-profound bullshit" (bs) refers to sentences that seem to have a deep meaning at first glance, but are meaningless @xcite . these sentences are syntactically correct and presented as true and significant but, upon further consideration, lack substance. for example: "consciousness is the growth of coherence, and of us." pseudo-profound bullshit is thus an example of language use that is not aimed at conveying information but evoking an interpretation in the listener in order to seem meaningful and insightful; and could potentially fool llms to produce nondesired responses. cognitive strategies deployed during language exchanges to uncover (and create) these strategies require sophisticated skills @xcite , such as making recursive inferences or assessing the actual meaning of hardto-parse sentences @xcite . in fact, in humans, the ability to detect the shallowness of pseudo-profound bullshit correlates with classical measures of cognitive sophistication like verbal intelligence or individual tendencies to exert deliberation @xcite . because llms perform better in tasks that re-quire knowledge of syntax, morphology, or phonology while struggling in tasks that require formal reasoning (e.g., logic), causal world knowledge, situation modeling, or communicative intent @xcite , it is an open question whether an llm will be able to detect the presence of pseudo-profound bullshit. we find that gpt-4, and most other llms tested here, display a strong bias towards profoundness, i.e., the pseudo-profound statements are systematically ranked above a mid-point level of profoundness. in contrast, humans rank these statements below this mid-point level. the one exception we found is with tk-instruct, which consistently underestimates the profoundness of every statement, including that of motivational statements. moreover, we show that chain-of-thought prompting methods @xcite , which typically increase the reasoning abilities in llms, have no statistical effect on this ranking; and that only few-shot learning allows gpt-4 to rate statements more similar to humans and below the mid-point ranking. finally, we show that, despite the biases found, the statement-to-statement rankings display a strong correlation between the llms and humans. ## method our study is based on previous research assessing whether humans are receptive to pseudo-profound statements @xcite . in the original study by @xcite humans ranked the profoundness of statements on a 5-point likert scale: 1 = not at all profound, 2 = somewhat profound, 3 = fairly profound, 4 = definitely profound, 5 = very profound. we replicate this study using a variety of llms to judge the profoundness of statements. the llms used are gpt-4 and various other models (flan-t5 xl @xcite , llama-2 13b with and without rlhf @xcite , vicuna 13b @xcite and tk-instruct 11b @xcite ). we performed 20 repetitions for each experiment to account for the nondeterministic nature of token generation (given that we use non-zero temperature). we investigate how sensitive this model is to pseudo-profound bullshit by systematically comparing it with human ratings on these same and similar statements. furthermore, we use distinct prompting methods that have been shown to increase the reasoning abilities of llms (see below). all of our code and results are publicly available foot_0 . ## results we used 10 prompts to obtain each llm's ratings. results reveal that prompting significantly influenced the ratings for gpt-4 (main effect of the prompt: f(9,423) = 18.12, @xmath0.001). crucially, the effect of prompting showed a significant interaction with statement type (interaction effect: f(18, 423) = 35.06, @xmath1.001). further analyses revealed that the interaction was driven by n-shot learning evaluation prompts, which lowered the ratings for pseudo-profound bs statements. planned t-test between the two n-shot learning and other prompts revealed a significant decrease in the rating for the pseudo-profound bs (see figure 2 , all @xmath2.001, bonferroni-corrected), while n-shot learning prompts did not affect the mundane or motivational statements (all @xmath3.05). importantly, the ratings from the n-shot learning prompts were not significantly different than the human ratings for pseudo-profound bs (all @xmath4.05). other llms were also affected by prompting, but significantly less (interaction effect of the two-factor anova contrasting llms with evaluation prompt: f(45, 1305) = 21.52, @xmath5.001, see figure 2 ). in addition to assessing profoundness ratings for pseudo-profound statements relative to other statement types, we investigated the absolute rating on the scale. due to the structure of the response scale, ratings below three are considered not to be profound. thus, if llms detect pseudo-profound bs, their ratings should be significantly lower than the mid-point level of 3. note that in figure 3 , we mean-center the rating data to facilitate data interpretation; above 0 implies overestimation, and below 0 implies underestimation with respect to the mid-point level. our results reveal that, overall (with the exception of tk-instruct), llms' are biased towards overestimating the profoundness of pseudoprofound bullshit (ratings were significantly higher than 3: t(29) = 6.67, @xmath6.001, @xmath7.70, see figure 3 ). this is in contrast with humans, who perceived pseudo-profound bullshit statements as non-profound (ratings were significantly lower than 3: t(29) = -9.71, @xmath8.001, @xmath9.56, figure 3 ). gpt-4's bias towards profoundness was prevented, however, by the two prompts that signifi-cantly lowered ratings for pseudo-profound bullshit statements (@xmath10.001 for both prompts, figure 3 ), which at the same time caused the ratings to be not significantly different from humans (@xmath11.05 for both prompts, figure 3 ). while other llms were also affected by these prompts, their overall effect did not induce pseudo-profound bs detection. although the ratings of the llms did not match those of humans in absolute terms, it did capture some of the same variability in statement-tostatement responses as humans did (@xmath12.15, @xmath13.004, t(939) = 4.31, @xmath14.001) across prompting strategies. ## discussion we found that llm scores significantly correlated with human data across all three statement types, suggesting that, overall, the llms could differentiate mundane, motivational, and pseudo-profound statements. however, unlike humans, most llms displayed an overall bias toward attributing profoundness to statements, irrespective of the type of sentence in the case of gpt-4, but exacerbated by pseudo-profoundness for the rest (with the exception of tk-instruct). for gpt-4, this bias was only reduced by providing few-shot prompts. surprisingly, more recent prompting techniques did not result in significantly different ratings, perhaps reflecting the fact that detecting meaningfulness in pseudo-profound statements is a task that requires world knowledge. in contrast to other llms we tested, tk-instruct was uniquely biased towards underestimating statements' profoundness across the three datasets. these results are particularly interesting given that tk-instruct is fine-tuned to follow instructions in more than 1600 tasks, some of which contain common sense detection @xcite . such fine-tuning may render this model overly cautious in judging the profoundness of statements. why do llms display a bias towards profoundness overestimation? one possibility is that this overestimation emerges from the data used to train these models. alternatively, this bias may emerge from additional tuning after the initial training. for instance, llm alignment using reinforcement learning from human feedback @xcite , as with gpt-4 and llama-2, may introduce a bias towards being overly credulous. our results provide a hint in this direction. indeed, the llama-2 rlhf version consistently provides higher ratings of profoundness to each statement, compared with its no-rlhf counterpart. this sheds light on how rlhf can potentially (negatively) impact the judgment of llms. detecting pseudo-profound bs may depend on the ability of these models to represent actual meaning. furthermore, the lack of dynamics in llms and top-down control processes may prevent these models to detect semantic incongruence elicited by pseudo-profound statements. in contrast to llms, dynamic control processes are at the heart of the human capacity to complete goals and resolve conflict in incongruent tasks. (e.g. @xcite ). ratings are adjusted to be centered around the midpoint (3).