Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? (2024)

Abstract: Despite the high performances of large language models (LLMs) across numerous benchmarks, recent research has unveiled their suffering from hallucinations and unfaithful reasoning. This work studies a type of hallucination induced by semantic associations. We investigate to what extent LLMs take shortcuts from certain keyword/entity biases in the prompt instead of following correct reasoning paths. To quantify this phenomenon, we propose a novel probing method and benchmark called EUREQA. EUREQA is an entity-searching task where a model finds a missing entity based on described multi-hop relations with other entities. These deliberately designed multi-hop relations create deceptive semantic associations, and models must stick to the correct reasoning path instead of incorrect shortcuts to find the correct answer.Experiments show that existing LLMs cannot follow correct reasoning paths and resist the attempt of greedy shortcuts, with GPT-4 only achieving 62% accuracy. Analyses provide further evidence that LLMs rely on semantic biases to solve the task instead of proper reasoning, questioning the validity and generalizability of current LLMs’ high performances.

## introduction recent progress of large language models (llms) has showcased their emergent abilities in a wide range of reasoning tasks, encompassing competence in program synthesis @xcite , mathematical reasoning @xcite , symbolic logic reasoning @xcite , and common-sense reasoning @xcite . innovations in inference methods have further improved language models' reasoning capabilities, either by generating intermediate steps toward the ultimate solution @xcite or by decomposing complex inquiries into manageable subproblems @xcite . research has further demonstrated that when incorporated with these techniques, llms have achieved exceptional results on multi-hop and complex qa benchmarks @xcite . nonetheless, it remains under-investigated whether the effectiveness of llms' reasoning pattern @xcite is genuinely based on a sensible reasoning path or if these models predominantly depend on semantic associations (i.e., word distributions from the pre-training data) to generate a plausible answer. the nutt house was written by alan spencer, mel brooks, and bruce bilson. therefore, person a could be alan spencer, mel brooks, or bruce bilson. figure 2 : even given all the required information needed for the question (selected information shown in the figure), gpt-4 still makes mistakes starting early layers (highlighted in grey). we only show partial output here. notice that we give gpt few-shot prompts. chatgpt 2 fails to find the missing entity as we gradually remove relevant semantic cues as the reasoning depth grows. the original question (labeled as "1 layer" in fig. 1 ) asks models to find an individual that satisfies two distinct conditions. it can be resolved by finding the set of people who are producers of the film "gardener of eden" and the set of cast members of "j. edgar", and checking for the overlapping entity. however, language models such as chatgpt may not necessarily perform this sensible entity-seeking process. this is because the correct answer can simply (but incorrectly) be inferred by the high occurrence of "leonardo di-caprio" in the model's training data with "gardener of eden" and "j. edgar". 3 in such a scenario, chat-gpt may generate the correct response through this shortcut of semantic association. using semantic shortcuts such as co-occurring entities works well on common test cases. still, they limit systems' generalizability and robustness on cases not aligning with pre-training distributions. this is evident if we remove entities directly relevant to the answer, such as "j. edgar", and replace them with recursive entity-seeking problems where models must first identify them, shown as "3 layers" in fig. 1 . we observe that chatgpt no longer produces the right answer without enough semantic 2 in this paper, we consistently utilize the gpt-3.5-turbo-0301 checkpoint for chatgpt. 3 in fact, if we prompt chatgpt with only three keywords "gardner of eden," "j. edgar" and "actor" and let it generate anything, the outcome is almost always "leonardo dicaprio." hints and hallucinates "adam sandler", which may be because "eden" is related "adam", and at the same time, both the writer and main character of "gardener of eden" is named "adam." in addition, chatgpt almost always comes up with "adam sandler" if asked for actors named "adam." to investigate and quantify whether llms can follow a correct reasoning path and resist the attempt to take "greedy shortcuts" during inference, we introduce eureqa (extending underlying reasoning chains in qa), a sophisticated multi-hop question answering dataset, meticulously designed for diminishing semantic associations and gauging the capability of llms in undertaking extensively chained reasoning processes. to create eu-reqa, we design a method that removes semantic shortcuts that will lead to the correct answer while adding "deceptive semantic cues" that are distracting and irrelevant to the correct answer. we also guarantee a simple and consistent reasoning path and only use common facts that should be memorized by the models. in this way, if an llm fails on such tasks, it directly suggests that the llm is taking incorrect shortcuts during inference. for instance, we modify the original question in fig. 1 by substituting "j. edgar" with a placeholder dubbed "film a" and supplementing a descriptive sentence about the film. our intent with this strategy is to lessen the model's reliance on direct information from the film name. we can automatically extend the depth of a reasoning chain by further replacing entities in the descriptive sentence about "film a" with corresponding abstract names. in particular, this generation method is developed on a knowledge base, ensuring that the information employed predates the training data of the examined llms. we then evaluate model performances on the eureqa benchmark. we show that state-of-theart llms, foot_0 are incapable of proper reasoning for instances in eureqa, with gpt-4 only achieving 62% to 64% accuracy in identifying common entities in wikipedia and chatgpt less than 40%. on the contrary, humans achieve near-perfect performances without much effort because of the simplicity and consistency of the gold reasoning path. we also show that gpt-4 performance strongly correlates with the semantic similarities between the gold answer and other entities mentioned in the question. these findings combined suggest that even the best language models, with detailed in-context-learning (icl) processes, still fall for deceptive semantic shortcuts and hence hallucinate and fail on eureqa. to summarize, the contribution of our work is three-fold. first, we propose a novel method for generating question-answering data with extended reasoning chains, which allows us to create deceptive semantic shortcuts. second, we propose eu-reqa, a qa dataset specifically designed for evaluating llms in scenarios with reduced or deceptive semantic associations. finally, our extensive experimental findings suggest that contemporary llms predominantly depend on semantic associations in question answering. as entities in question are replaced with alternate inquiries, llms cannot adhere to an accurate reasoning process necessary for problem resolution. ## eureqa in this section, we introduce the dataset eureqa for evaluating llms on extended reasoning chains. we start with the definition and generation approach of a reasoning chain ( §2.1), followed by the processes for question generation ( §2.2) and refinement ( §2.3). ## experiment setup this section presents the experiment configurations employed for assessing the long-chain reasoning capabilities of llms through eureqa. this section begins with a discussion on model configurations ( §3.1), subsequently introducing the adopted prompting methodologies ( §3.2), and finally, it scrutinizes the evaluation configurations ( §3.3). ## results the results of the state-of-the-art llms on eu-reqa, as reported in tab. 2, illustrate the variations in performance across different depths of reasoning and levels of difficulty. in general, with the entities recursively substituted by the descriptions of reasoning chaining layers, and therefore eliminating surface-level semantic cues, these models generate more incorrect answers. when the reasoning depth increases from one to five on hard questions, there is a notable decline in performance for all models, with an average accuracy decrease of 14.2% when using the icl prompt and 14.3% when using the direct prompt. the performance of llms is significantly higher on the easy set compared to the hard set. this is evidenced by a marked increase in accuracy, averaging 7.0% for chatgpt, 15.4% for gemini-pro, and 19.7% for gpt-4. this finding underscores the significant impact that semantic shortcuts have on the accuracy of responses, and it also indicates that gpt-4 is considerably more capable of identifying and taking advantage of these shortcuts. it can also be inferred from the results that the demonstrated human-written examples do help improve the performance. for both chatgpt and gpt-4, using the icl prompt consistently leads to better performance than the direct prompt. this suggests that these models can benefit from examples that provide explicit and well-crafted humanwritten reasoning processes. these few-show examples encourage the models to resolve the questions with the correct reason instead of guessing with semantic cues. nonetheless, even with clear reasoning processes provided, models still tend to fail for incorrect shortcuts. another observation is that llms do not always perform worse on questions with more depth of reasoning. for instance, chatgpt with direct prompt is 3.5% more accurate when the depth of reasoning d increases from four to five. we believe this also indicates that llms do not always follow the reasoning path and may instead answer the question based on surface-level semantic shortcuts. further analyses can be found at §5.2 to identify the errors in these llm responses, we randomly sampled 20 data points where gpt-4 with icl, the best performing method in this benchmark, has generated incorrect answers. the most frequent error pattern lies in hallucination during the reasoning process, encompassing 80% of the cases. in such cases, an intermediate reasoning stage erroneously conceived an inaccurate, albeit plausible answer, culminating in accumulated errors in the final answer. for instance, the "founder of a rocket company" was misconstrued as "elon musk, the founder of spacex", rather than the correct answer of "jeff bezos, the founder of blue origin." this phenomenon proves our claim that llm relies on semantic cues for problem-solving instead of cognitive reasoning. ## analysis and discussions in this section, we perform further analyses and provide a more in-depth discussion of our findings. ## related work we discuss two topics of works that are highly relevant to this study. hallucination. existing research on analyzing llms hallucination mainly focuses on three aspects: input-conflicting hallucination, contextconflicting hallucination, and fact-conflicting hallucination @xcite . input-conflicting hallucination happens when the llms outputs are divergent from prompts @xcite . selfcontradictory outputs can occur in long-form or multi-turn answers, implying that llms lose track of the core input information during generation @xcite . factual-based errors -misaligns with established world knowledge -are most widely studied since they can happen in various llms to affect multiple tasks' performance @xcite . knowledge conflicts also lead to hallucination when in-context information contradicts what llms memorize from preraining @xcite @xcite . however, none of these existing works look into hallucination on reasoningrequired questions and test whether the models are following the correct reasoning path versus the semantic token bias. reasoning capability of llms. the success of in-context learning @xcite and instruction tuning @xcite have inspired various works to solve reasoning tasks by prompting llms. more advanced prompting strategies have been proposed to enhance the reasoning capabilities of llms. these investigations take advantage of the heuristic nature of human problem-solving procedures and incorporate them into textual prompts as guidance for the models. an example of this can be seen in chain-of-thought prompting @xcite , a method in which intermediate steps are generated leading up to a final solution. @xcite and @xcite proposed to decompose the questions into simpler, manageable sub-problems as a method to facilitate complex reasoning. researchers also attempted to investigate llms' reasoning ability over structured knowledge @xcite which necessitates reasoning with uncertainty as well as back-tracking. studies prior to llms also propose to generate rationales to improve with more faithful reasoning @xcite . nonetheless, this work questions the exhibited "reasoning capabilities" of llms by distinguishing them from the attempt to take greedy semantic shortcuts during the reasoning process. ## conclusion this paper proposes a novel qa benchmark for probing llm hallucinations induced by semantic shortcuts on reasoning chains. we introduced a systematic method for generating question-answering data with extended reasoning chains. this dataset enables us to examine the reasoning capabilities of llms on extended reasoning paths and our experimental results indicate that llms predominantly depend on semantic shortcuts for reasoning and such behavior contributes to its failure. our analyses questioned the validity of current llms and also inspired future studies. the problem-solving or reasoning trajectories of humans could be incorporated into the training or inference phase of current language modeling systems for more effective computational models.