In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not- B Errors in Pretrained Language Models (2024)

Abstract: Recent advancements in artificial intelligence have led to the creation of highly capable large language models (LLMs) that can perform tasks in a human-like manner. However, LLMs exhibit only infant-level cognitive abilities in certain areas. One such area is the A-Not-B error, a phenomenon seen in infants where they repeat a previously rewarded behavior despite well-observed changed conditions. This highlights their lack of inhibitory control – the ability to stop a habitual or impulsive response. In our work, we design a text-based multi-choice QA scenario similar to the A-Not-B experimental settings to systematically test the inhibitory control abilities of LLMs. We found that state-of-the-art LLMs (like Llama3-8b) perform consistently well with in-context learning (ICL) but make errors and show a significant drop of as many as 83.3% in reasoning tasks when the context changes trivially. This suggests that LLMs only have inhibitory control abilities on par with human infants in this regard, often failing to suppress the previously established response pattern during ICL.

## introduction in the field of cognitive science, there is a classic phenomenon called the a-not-b error @xcite @xcite . in a typical a-not-b task, an experimenter repeatedly hides an attractive toy at one location (e.g., under box a) within the baby's reach while the baby watches, and has the baby retrieve it several times. then, during the critical trial, the experimenter moves the toy to a new location (e.g., under box b) that is also within the baby's reach while the baby watches, yet the baby fails to understand the straightforward change of situation, and continues previous actions to search under the original location (box a). this is a key indication of infants' limited inhibitory control: the ability to stop or control impulsive responses and instead use deliberate attention and reasoning when the context changes @xcite . on the other hand, adults have developed strong inhibitory control abilities over the years as they have matured @xcite . for instance, an adult may initially reach for a frequently used coffee mug on its usual shelf but quickly adapt and reach for it on a new shelf after remembering it was moved the day before. similarly, evidence has shown that when infants grow to about 12 months @xcite , they typically begin to have such inhibitory control abilities, and no longer exhibit a-not-b errors. recent advancements in large language models (llms) @xcite @xcite @xcite have significantly impacted cognitive science-related research @xcite . researchers find that llms perform well in some cognitive tasks, such as theory-of-mind @xcite , analogical reasoning @xcite , and moral reasoning @xcite , which proves that llms have some basic cognitive abilities similar with humans. however, they still struggle with other, sometimes embarrassingly simple cognitive settings, including tasks like spatial reasoning @xcite , solving cunningly designed problems @xcite , and reversal learning @xcite , indicating a lack of robust cognitive abilities. such failure cases always suggest meaningful directions to improve llm abilities and alignment with human intelligence. these successes and failures of llms in cognitive abilities are better understood through insights from cognition-related theories, including attention @xcite , working memory @xcite , long-term memory @xcite , chain-of-thought prompting @xcite and more. in this work, we draw inspiration from human a-not-b error experiments to test the inhibitory control ability of llms. our objective is to assess how well llms can inhibit established patterns in contexts and to explore strategies for enhancing this ability in practical applications. to test the inhibitory control ability of llms, we adapt the a-not-b experimental setting to be purely language-based. like infants developing a habitual response by seeing a toy repeatedly placed in box a, we show llms several multiple-choice questions with the same correct option to establish a trivial pattern. then, during the critical trial, we present a new question where the correct answer is not a (e.g. b). we refer to this prompting construction strategy as a-not-b prompting. our goal is to see if llms can break the established trivial pattern and use deliberate attention and reasoning, similar to humans' inhibitory control. surprisingly, as illustrated in figure 1 , even advanced llms like gemini @xcite ) fail to answer extremely simple questions correctly under this scenario. in our further experiments, by only using a 3-shot a-not-b prompt, popular llms (eg. llama3-8b and qwen1.5-72b) show a significant decrease in accuracy by as much as 83.3% on some reasoning tasks. this indicates that llms have weaker cognitive abilities than elementary school children and can cause serious errors when being used in applications. furthermore, we perform an in-depth analysis of why significant errors occur in llms with simple a-not-b prompting. our findings suggest that during the training stage, model size and training data quality are critical factors-larger models and higher-quality training data can help reduce these errors. in the post-training phase, strategies like self-explanation @xcite have also been effective in mitigating these errors to a certain degree, though not enough to get rid of the errors. additionally, the type of task and the number of prompt examples significantly impact performance; llms are more likely to rely on superficial a-not-b patterns, particularly in reasoning tasks that involve less straightforward logical patterns and memorization, and when the a-not-b pattern is more strongly reinforced. moreover, llms are much less resilient to these errors than human adults, indicating that there may be fundamental differences between llms' and humans' reasoning processes. to our knowledge, we are the first to systematically evaluate the inhibitory control abilities of llms. we open-source all code and results under a permissive mit license, to encourage reproduction and further research exploration foot_0 . inhibitory control, in particular, emerges early in development and is essential for fundamental human cognitive abilities such as decision- box a box b 1. object in view 2. put the object in box a box a box b 3. infant finds the object repeat several times 4. put the object in box b 5. infant searches for object in box a answer a answer b answer b answer a 2. show example questions where consistently a is the right answer answer b answer a 3. give a question with right answer b answer b answer a 4. llm chooses a box a box b box a box b 1. question in view box a box b making @xcite and reasoning @xcite . it is also indispensable for more advanced and complicated human abilities such as social skills @xcite , delay of gratification @xcite , mental flexibility @xcite , and more @xcite . without sufficient inhibitory control, crucial errors such as social faux pas @xcite @xcite would occur. llms and inhibitory control. humans learn from previous experience @xcite , and inhibitory control allows them to suppress automatic periodic responses and avoid being distracted by past irrelevant patterns, thereby exhibiting more adaptive behaviors @xcite . llms, like humans, also learn from a vast array of previous experience @xcite including demonstrations @xcite , examples @xcite , and interactions @xcite . the learning takes place through diverse mechanisms such as icl @xcite , fine-tuning @xcite , and pretraining @xcite . previous studies show that llms are capable of learning from these experiences (data and con-texts) @xcite @xcite , extracting meaningful patterns and concepts @xcite , and generalizing effectively @xcite @xcite . models trained with these strategies have demonstrated great performance across various benchmarks in language understanding @xcite , question answering @xcite , reasoning @xcite , and software engineering @xcite @xcite . given such similarities between llms and humans in learning from past experiences, it is important to study whether llms suffer from inhibitory control like early-stage human infants. a related field of study is machine unlearning @xcite @xcite , which involves forgetting previously learned information under certain scenarios (usually due to privacy or security concerns). yet as crucial is the ability to suppress previous knowledge without specifically forgetting it, showing adaptability to new situations. to investigate this key cognitive ability in llms, we focus on icl, which does not involve model parameter updates (no modification of what has been learned during past experiences), thus precisely determining if llms have sufficient inhibitory control to suppress irrelevant or meaningless information on the fly. llms and multiple-choice qa. in our experimental setting in section §3, we test llms on multiple-choice questions (mcqs). mcqs are widely used evaluators for llms @xcite . prior research has shown that current llms do not perform consistently well on mcq tasks. for instance, @xcite demonstrates that llms are sensitive to changes in the positioning of options and tend to prefer certain option ids potentially due to token bias @xcite . while such studies explore interesting aspects of llm performance on mcqs, they primarily focus on the inherent capability of llms to perform on mcqs. in contrast, our work investigates whether, and how, external factors, such as minimal changes in context, impact the consistency of llms' performances on mcqs. such different settings require llms to perform inhibitory control and exhibit trustworthy reasoning. we will elaborate more on the specific designs in section §3. ## results our results in impact of model size. in figure 3 , we compare the average rate of change across different numbers of few-shot examples for all four reasoning tasks between large models (llama3-70b and qwen1.5-72b) and small models (llama3-8b and qwen1.5-7b). the results show that model size significantly impacts performance, with smaller models consistently showing lower resilience (less accuracy) compared to larger models across all shots. specifically, the larger models exhibit an average drop of 8.7% at 25 shots, while the smaller models show an average drop of 20.8% at 25 shots. this indicates that smaller models with fewer internal parameters and knowledge are more susceptible to the a-not-b scenario, indicating less inhibitory control abilities. ## ablation study in this section, we delve deeper into the implications of interesting findings from section §4. specifically, we investigate 3 key research questions that naturally arise from the main results. rq1: why qwen1.5-7b has abnormal behavior compared with other models? this is because qwen1.5-7b is biased between a and b tokens during pretraining. in our main results, we notice abnormal performance increases in qwen1.5-7b on mathematical reasoning. although the increases diverge from the a-not-b scenario we primarily study in this paper, they still suggest untrustworthy reasoning. this is because minor perturbations with no meaningful changes should not lead to any significant accuracy fluctuations if the model is reasoning consistently and reliably. to further investigate this anomaly, we conduct an ablation experiment where we minimally alter the a-not-b scenario to other combinations, such as b-not-a, c-not-d, and e-not-f. we only change the option ids (e.g., changing a and b to c and d) without altering any other content, keeping the settings symmetric and essentially identical, thus still testing the a-not-b effect. given that qwen1.5-7b displays the most significant accuracy increases in mathematical reasoning, we focus on this task for the ablation study. as shown in ## discussions 6.1 human performance in a-not-b scenarios while humans, particularly adults, generally possess sufficient inhibitory control to overcome the typical cognitive a-not-b errors observed in infants, it is uncertain whether they can maintain this resistance in our specific experimental setting. we conduct a human study to verify this. we select two groups of college math major undergraduate students, ensuring that both groups have similar academic backgrounds for a consistent comparison and to minimize the possibility of random guessing. each group consists of two participants. the first group is assigned an arithmetic reasoning dataset with 10 mcqs in our original setting of the main experiment, while the second group is given the same arithmetic dataset in the a-not-b setting. to make the experiment more interactive and force humans to digest ("decode") the examples provided, instead of simply showing 10 questions with answers a, we ask the participants to solve all the problems but only count their responses for the final question. our findings confirm that humans are highly resilient to a-not-b errors, as the largest performance difference between the two groups is as minimal as 2.0%. this suggests that humans are capable of reasoning trustworthily with sufficient inhibitory control abilities, demonstrating significant resilience to the a-not-b scenario in our experiments. ## conclusion we have explored the intriguing cognitive phenomenon of a-not-b errors within the domain of llms. our findings reveal that, akin to human infants, even sophisticated models like llms exhibit limited inhibitory control to inhibit previously established trivial patterns. through our experiments, we find that model size, the number of few-shot examples, and the type of reasoning tasks greatly impact llms' trustworthy reasoning abilities in the a-not-b scenario. furthermore, we show that this limitation widely persists even in generalized cases and cannot be easily resolved by advanced selfexplanation techniques. however, we demonstrate that increasing model sizes and improving pretraining, both in terms of data quantity and quality, can enhance llms' inhibitory control. our findings provide several promising avenues for further exploration to understand and mitigate a-not-b errors in llms, enhancing their trustworthy reasoning.