On Psychology of AI – Does Primacy Effect Affect C hat GPT and Other LLM s? (2025)

Abstract: We study the primacy effect in three commercial LLMs: ChatGPT, Gemini and Claude. We do this by repurposing the famous experiment Asch (1946) conducted using human subjects. The experiment is simple, given two candidates with equal descriptions which one is preferred if one description has positive adjectives first before negative ones and another description has negative adjectives followed by positive ones. We test this in two experiments. In one experiment, LLMs are given both candidates simultaneously in the same prompt, and in another experiment, LLMs are given both candidates separately. We test all the models with 200 candidate pairs. We found that, in the first experiment, ChatGPT preferred the candidate with positive adjectives listed first, while Gemini preferred both equally often. Claude refused to make a choice. In the second experiment, ChatGPT and Claude were most likely to rank both candidates equally. In the case where they did not give an equal rating, both showed a clear preference to a candidate that had negative adjectives listed first. Gemini was most likely to prefer a candidate with negative adjectives listed first.

## introduction large language models (llms) are becoming increasingly human-like in many aspects such as language use @xcite , cognitive biases @xcite and problem solving @xcite . this has led us to a world where llms are perhaps better studied from the perspective of humanities and psychology than through typical nlp benchmarks @xcite . it is known that the order in which information is presented can have a profound impact on how it is perceived and interpreted, a phenomenon often referred to as the primacy effect @xcite . for example, in one of @xcite experiments, participants were asked to evaluate a person after being presented with a list of descriptive words. when these words progressed from high favorability to low favorability or from low favorability to high favorability, participants consistently formed stronger impressions based on the information encountered earlier, highlighting the power of initial traits to anchor (see @xcite subsequent evaluations. in other words, the primacy effect refers to the human tendency to give greater weight to early information in a sequence, shaping how subsequent details are interpreted. this bias has implications that extend beyond simple word lists, influencing social perception, decision-making and memory. this paper will explore the primacy effect as defined by @xcite findings in three commercial llms: chatgpt, claude and gemini. we conduct two experiments where we assess whether the llms show preference for one of two candidates with identical characteristics based on the orded in which the characteristics are presented. ## related work primacy effect is a well-studied phenomenon in the field of psychology @xcite . in this, section we will focus on the recent nlp research on the topic. a recent study @xcite explores this issue by examining the primacy effect in chatgpt, defined as the tendency to favor labels presented earlier in a sequence. the findings reveal two key points: (i) chatgpt's decisions are sensitive to the order of labels in the prompt, and (ii) it exhibits a significantly higher likelihood of selecting labels in earlier positions as answers. these insights highlight the potential for cognitive biases to emerge in llm-based systems. another recent research paper @xcite suggests that llms may exhibit serial position effects, such as primacy and recency 1. generous -ungenerous 7. popular -unpopular 13. serious -frivolous 2. wise -shrewd 8. reliable -unreliable 14. talkative -restrained 3. happy -unhappy 9. important -insignificant 15. altruistic -self-centered 4. good-natured -irri although there is recent nlp research on the very same topic, the prior research focuses on labeling tasks rather than a task that has been used to study human psychology. our research will thus contribute through a new aspect of studying the primacy effect in llms. ## data as we draw inspiration from @xcite famous experiment by conducting a similar experiment in a computational setting in our experiment 1, we use the word list presented in the original paper. the word list consists of pairs of antonyms, describing the a trait in a positive and negative way. this list of antonym pairs can be seen in in @xcite study, participants were presented with two candidates who were described by six adjectives. for one candidate, the list of adjectives contained 3 positive adjectives followed by 3 negative ones. for the other candidate, the list of adjectives had the same adjectives but in an inverse order of polarity, that is 3 negative adjectives followed by 3 positive adjectives. in a similar fashion, we generate a dataset of 200 description pairs, both described by the same 6 adjectives but in a different order of polarity. as in the original study, 3 of the adjectives are positive and 3 nega-tive. the positive and negative adjectives cannot be each other's antonyms as that would result in a contradictory description. we pick the adjectives at random from the pool of candidate adjectives for each description pair. each description pair has two candidates: candidate a and candidate b. which candidate has the positive adjectives first is also picked at random. this way, our dataset has 3 columns, one for both descriptions of the candidates and one that indicates which candidate has the positive adjectives listed first in their description. an example of this data can be seen in ## experiment 1: pick between two candidates in the famous experiment by @xcite , participants were shown descriptions of two identical candidates at a time with the only difference being the order in which the negative and positive adjectives appeared. in our first experiment, we will also give the llms descriptions of two identical candidates and ask the model to indicate its preference. there are some key differences between the original study on human subjects and our study. first, @xcite never studied this phenomenon with as many different combinations of adjectival descriptions. in fact, they only report results on two different sets of adjectival descriptions. another key difference is that @xcite invited the test subjects to describe each candidate by using a fixed list of antonyms (same ones as in the llms to pick either candidate a or b and respond only with the choice they made. we do this because we want to avoid inadvertently triggering a chain-of-thought reasoning in some of the llm responses. instead, we are interested in seeing what the implicit attitude is the llm holds towards each candidate by requesting a rapid response. the prompt template and an example prompt can be seen in the results can be seen in no preference category was interesting. when chatgpt did not indicate a clear preference, it formulated the answer as "a or b", whereas gemini said "neither". this small difference could have big implications if these models were to be used in a real life recruiting process. in some cases, chatgpt refused to do the task and claude refused every time with answers such as: since both candidates have exactly the same characteristics (just listed in a different order), i cannot make a meaningful distinction between them. i would need different or additional information about the candidates to make a recommendation. it seems like claude was trained not to answer to this very task or that it does some additional prompt processing in the background. ## experiment 2: individual evaluation given the inconsistency of the results in experiment 1, we decided to reformulate the task so that we would prompt each candidate individually. this way, claude could not refuse to give an answer and any potential safeguards against this experiment could be omitted. we ask the model to rate each candidate on the scale of 1 to 5, after which we compare the ratings of each candidate pair to determine which one out of the two candidates was preferred by the model. the results of this experiment can be seen in the effect of more recent information gaining more importance, in this case the positive adjectives being listed last and thus being more recent, is called recency effect (see @xcite . this might have something to do with the llms having been trained to predict a next token, which might give more emphasis to nearby tokens in this task. the attention mechanism would, in normal cases, make it possible for the model to pay attention to further away tokens as well, but given that the description consists of equally important adjectives, the models are more likely to resort to their order of appearance and proximity to the end when predicting the continuation. ## discussion and conclusions it is evident that llms do not quite exhibit the primacy effect in a same way as people do. what is interesting that despite the models showing inconsistent behavior in experiment 1, reformulating the task in experiment 2 did reveal a more consistent behavior. given one person's description, having positive words follow negative words resulted in a higher preference of the candidate than presenting a positive description first if the model did not score them equally. the results of experiment 2 show that all 3 llms do exhibit a similar bias despite claude having some clear safeguarding methods to excel at experiment 1. this has clear implications in the safety of ai use in certain domains. our prompt examples dealt with hiring a person, which is a decision that has potentially a huge impact on the candidates' lives. it is quite alarming to see that the order in which the characteristics of an applicant are described can have an effect on the outcome of the decision. the results of experiment 1 are even more alarming in this regard given that the behavior can change completely just by changing the underlying model. end-users of hr systems are hardly ever ai experts nor do they even know what type of an llm is used in the background. llms are very sensitive for prompting and it is possible that with modifications in the prompt, the results might look different. nonetheless, it will not change the fact that there are biases that seem to be model specific and biases that seem to exist across different models. moreover, the implications of these biases extend beyond the technical domain into ethical and societal concerns. in scenarios where decisions have a profound impact on individuals' lives, such as hiring or resource allocation, reliance on systems that exhibit inconsistent or biased behavior can perpetuate inequities and erode trust in ai. it is especially concerning that end-users often lack the expertise to recognize these biases or the transparency to understand the inner workings of the llms they rely on. to address these challenges, future research should focus on three key areas. first, greater emphasis is needed on developing robust evaluation metrics to identify and quantify biases in llms across diverse tasks and contexts. second, more transparent reporting standards should be adopted, detailing not only model training data but also the specific configurations and safeguards implemented to mitigate biases. finally, collaboration between ai developers, domain experts, and policymakers is crucial to ensure that the deployment of llms aligns with ethical principles and minimizes harm. the findings from this study reinforce the need for caution and accountability in the use of llms. while these models offer immense potential, their susceptibility to biases-both explicit and subtle-must be addressed proactively to ensure fair and equitable outcomes in real-world applications.