Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination (2024)

Abstract: This paper presents a series of investigations into an interesting phenomenon where we observe performance increases in large language models (LLMs) when providing a prompt that causes and exploits hallucination. We propose null-shot prompting, a counter-intuitive approach where we intentionally instruct LLMs to look at and utilize information from a null section. We investigate null-shot prompting on a wide range of tasks, including arithmetic reasoning, commonsense reasoning, and reading comprehension. We observe a substantial increase in performance in arithmetic reasoning tasks for various models, with up to a 44.62% increase compared to a baseline in one model. Therefore, we investigate deeper into this task by utilizing a more challenging mathematics problem-solving benchmark. We observe that LLMs benefit from hallucination in null-shot prompting in this task and discuss the mathematical topics that benefit the most from introducing hallucination in the prompt. We continue our investigation by evaluating hallucination detection abilities of the LLMs when using null-shot prompting. We find surprising results where hallucination in prompts can improve hallucination detection abilities of many LLMs. We also examine the effects of introducing both reasoning, which is known to mitigate hallucination, and hallucination simultaneously in the prompt and observe another surprising turn for the mathematics problem-solving benchmark with many performance improvements. We hope this paper will spark more interest, investigations, and discussions on how hallucination in prompts LLMs and even bolsters them in certain cases.

## introduction hallucination of generative models, in a broad sense, is defined as a situation where there is conflicting information, either with facts, established knowledge, intents, or previously generated or provided content, within their context window @xcite @xcite . hallucination existed even before the recent widespread usage of large language models (llms) @xcite . however, it has become an increasing problem given the integration of llms in applications of growing importance @xcite @xcite @xcite . therefore, previous studies have extensively explored approaches ## null-shot prompting we propose a null-shot phrase for null-shot prompting suitable for general tasks, as presented in figure 2. this phrase is placed at the beginning of the prompt. the decision to position the phrase at the beginning is due to better performance compared to placing it at the end of the prompt, as demonstrated in section 7.1. the original optimized prompt for text toxicity classification containing the hallucination inspiring our null-shot prompt is provided in section b. we acknowledge that llms, when encountering such scenarios, should instead refuse the request or ask for the missing information, as expected behaviors, instead of simply following such hallucinatory prompts. we discuss cases from our experiments where we observed such behaviors from certain llms in section f.3. ## evaluation of null-shot prompting we select a broader set of tasks for the evaluation of null-shot prompting on tasks commonly used for evaluating the performance of llms. these tasks consist of arithmetic reasoning @xcite and @xcite ), commonsense reasoning @xcite and winogrande @xcite ), reading comprehension @xcite ), natural language inference (anli @xcite ), and closed-book question answering (triviaqa @xcite ). we also select a comprehensive list of llms from various model families to provide a complete picture, consisting of palm 2 (text and chat generation) @xcite , gemini 1.0 pro (text and chat generation) @xcite , gpt-3.5 turbo, and gpt-4 turbo @xcite . we include claude models (anthropic, 2024), specifically claude 2.1, claude 3 haiku, claude 3 sonnet, and claude 3 opus, as part of the evaluation, as they are well-known for safety alignment, i.e., less hallucinatory, to better understand how this aspect affects null-shot prompting, which causes and exploits hallucination. a more detailed description, rationales, and evaluation approach are provided in appendix c. we report the performance of relative changes when using null-shot prompting compared to the zero-shot prompting baseline in we observe that null-shot prompting is able to improve the performance of palm 2, both text and chat generation. we observe great improvement in arithmetic reasoning in gemini 1.0 pro, both text and chat generation, as well as gpt-3.5 turbo. for state-of-the-art models, gpt-4 turbo and claude 3 opus, we observe a similar trend of performance improvement in reading comprehension. however, in the majority of cases for claude models and gpt-4 turbo, we observe a subpar performance when using null-shot prompting. palm 2 and gemini 1.0 pro, both developed by google, retain the effectiveness of null-shot prompting despite following different training recipes. however, we note one difference between these models observed in our experiments: we do not observe performance gains from null-shot prompting in commonsense reasoning tasks and closed-book question answering. we argue that this is due to the alignment process present in gemini 1.0 pro and not in palm 2, which emphasizes not hallucinating incorrect information in closedbook question answering scenarios @xcite and commonsense reasoning tasks, which elicit reasoning in llms. therefore, reducing the chances of hallucination through improved recall and reasoning @xcite . in fact, commonsense reasoning, along with closed-book reasoning, are tasks where we observe the least number of models benefiting from nullshot prompting. we discuss more on reasoning and null-shot prompting in section 6. however, we note that gemini 1.0 pro, which was emphasized to perform hedging when encountering scenarios similar to the null-shot phrase, as written in their report @xcite , is ineffective in most scenarios as can be observed from the increased performance. contrary to previous observations, we notice that llms, which are currently among the strong models considered to achieve state-of-the-art performance across many tasks, like gpt-4 turbo and claude 3 opus, generally lose the effectiveness of null-shot prompting, except in the case of the reading comprehension task. in fact, null-shot prompting is able to make gpt-4 turbo achieve the best performance in race-m and race-h among the evaluated models. one unique aspect that only exists within this task, and not in other tasks, is its long context nature. this long context, due to including a long article, leads llms to hallucinate and think that the provided article is the mentioned section. thus, the null-shot phrase acts as additional conditional tokens reinforcing the llms to pay attention to the provided article in this case, leading to better performance. nevertheless, we acknowledge that the minimal increase in gpt-4 turbo could also be attributed to normal prompt sensitivity. on the other hand, we observe that null-shot prompting does not elicit the same trend of performance change for these state-of-the-art llms in the closed-book question answering task @xcite . this task shares one similarity with reading comprehension in requiring the model to recall certain information, but the difference is that race requires the model to recall information within the context, while triviaqa requires the model to recall information from their own parameters. the null-shot phrase instructs these llms to look and utilize information from the null section, which is also a form of instruction focusing on information recall, albeit such information never exists within the context. it shows that for these state-of-the-art llms, null-shot prompting does not elicit the models to recall information from their parameters but is instead effective for recalling within context as previously discussed. claude models are known for their emphasis on safety and harmlessness, as evidenced by prior work from their developers @xcite @xcite . therefore, we expect null-shot prompting to become less effective and unsurprisingly, the results observed match our expectations. furthermore, we observe a higher decrease in performance in the commonsense reasoning task, further supporting the argument that reasoning decreases hallucination as known in the community. however, we observe that the stateof-the-art model, claude 3 opus, benefits from null-shot prompting in the reading comprehension task, as previously discussed. furthermore, claude 3 haiku, the smallest model in the claude 3 series, has high performance gains from null-shot prompting in the natural language inference (nli) task. since it is a proprietary model, we cannot say for certain on which changes in the model are responsible for the observed effects. based on the overall results so far, we establish a few observations: another crucial observation is performance gain in arithmetic reasoning tasks where we observe high performance improvement in palm 2, gemini 1.0 pro, and gpt-3.5 turbo, especially gemini 1.0 pro and gpt-3.5 turbo. these tasks, at first, seem unrelated to and likely unbeneficial from the hallucinatory instruction in the null-shot phrase, as they focus on reasoning in computation steps and numerical calculations. however, when considering from the perspective that hallucination is a way of expressing creativity in llms, we believe that this kind of task will benefit more, as arithmetic reasoning, unlike other tasks, has numerous ways to reach the final answer and often requires creativity in devising such intermediate steps. therefore, it is likely that these llms also benefit from such creativity from the hallucination in null-shot prompting. we investigate this task further in the next section. ## math evaluation sets we expand our evaluation by focusing on the arithmetic abilities of llms, as we previously observed significant improvements when using null-shot prompting for certain llms in arithmetic reasoning tasks. we select the math dataset @xcite , which not only offers more challenging questions but also covers a variety of topics, aiding in further analysis. we follow the evaluation methodology of the previous section, with the addition of instructions for the model to output in a specified format to aid in the evaluation process. this format follows the original dataset labeling. the results are presented in the results for palm 2 (chat) show significant improvements across topics, with the most substantial performance improvement in algebra. however, we do not observe the same trend for palm 2. on the other hand, gemini 1.0 pro and gemini 1.0 pro (chat) share the same trend of performance changes. aside from performance improvements across most topics, we observe an interesting insight in the number theory topic where null-shot prompting causes no change in performance, which is surprising as changes in prompts usually lead to differences in the outcomes of the models. the not only are there occurrences of no changes in performance, but we also observe that in cases where null-shot prompting causes a performance decrease, it is less deviated from the zero-shot performance baseline compared to the previous section's evaluation. this is especially true for claude models, where we see less performance decrease and even observe some performance gains in models and topics. this leads us to believe that mathrelated tasks may require a certain degree of hallucination or creativity to perform well. counting & probability and intermediate algebra are two tasks that show the highest number of llms benefiting from null-shot prompting. this indicates that null-shot prompting is effective in problems requiring statistical and symbolic reasoning. in contrast, only a moderate number of llms benefit in prealgebra and algebra; these tasks focus more on numerical calculations. we also note that gpt-3.5 turbo, another llm that benefits from null-shot prompting for math problem-solving, shows high performance gains across topics, with less notable increases only in precalculus. gpt-3.5 turbo and palm 2, which benefit the most from null-shot prompting, are based on decoder-only transformer architecture. we believe that null-shot prompting requires models to use transformer architecture and undergo chat-tuning to be creative in math problem-solving and exhibit the performance gain. ## hallucination detection since null-shot prompting includes hallucinatory instructions, we explore how it affects the hallucination detection abilities of llms. we hypothesize that by including hallucination in the prompt, mod-els will suffer from degraded abilities in hallucination detection, since there are conflicting elements in the prompt, namely, hallucination. to evaluate hallucination detection, we utilize halueval @xcite , a hallucination detection dataset which contains scenarios such as general dialogue, question answering, and summarization. we adapt the original prompts from the evaluation set to suit our task by removing few-shot examples in the prompts and evaluating in zero-shot or null-shot scenarios instead, to reduce factors affecting performance during the analysis. we present results from halueval in these results are quite surprising as they contradict our hypothesis and are very counter-intuitive in the sense that providing a prompt with hallucination improves the hallucination detection abilities of the llms. one similarity in the results of palm 2 (chat) from the math evaluation and halueval evaluation is that this model is relatively weak in these evaluations compared to the other llms. in for dialogue scenarios, we see that all models, except for claude 2.1 and claude 3 opus, have increased performance when using null-shot prompting, supporting our previous discussion about the chat-tuned version of palm 2 gaining the most increase in performance. similar to previous observations of other evaluation sets, we also see gpt-3.5 turbo gaining significant performance increases in dialogue and question answering scenarios. as previously discussed, this is another evidence that null-shot prompting is effective with chat-tuned transformer models. additional studies and analyses are provided in the appendices. ## reasoning and hallucination in this section, we investigate how reasoning, known to reduce hallucination, combined with hallucination in prompts, can affect the llms. we follow the same setups as sections 3 and 4. however, we change the baseline to zero-shot chain-ofthought (0cot) prompting @xcite and compare it against ∅cot. ∅cot combines the original null-shot prompting with a phrase from 0cot instructing an llm to think "step-by-step" (reasoning). we believe that this contrasting instruction proves to be interesting to observe and may help shed some light on understanding the inner workings of llms. we present the results of the evaluations in however, there are also cases where a performance increase in the reading comprehension task is very noticeable in palm 2 and claude models, except claude 3 opus. for example, claude 3 haiku achieves a 44.46% and 36.42% performance increase for race-m and race-h, respectively, compared to a strong 0cot prompting baseline. the reason behind this is likely as previously discussed regarding the long-context nature of the task and the possibility of changes in architecture for state-of-the-art llms. for the datasets utilized in the main experiment, 19 combinations show improvements, 11 of which are statistically significant: four demonstrate medium effect sizes, four show small effect sizes, and three exhibit large effect sizes. on the other hand, there are 40 improved combinations for the math datasets, 10 of which are statistically significant: seven demonstrate medium effect sizes, and three show small effect sizes. we also observe cases where some llms gain substantial performance increase when using ∅cot prompting. gemini 1.0 pro models for arith- for the math evaluation, we observe a different trend compared to the general evaluation sets. we find that null-shot prompting exhibits higher effectiveness, a trend in the same direction as when we evaluated null-shot prompting for the math dataset in section 4. this is quite surprising given the fact that we observe mostly no performance improvement in aqua and gsm8k when using ∅cot prompting, which are also mathematicsrelated tasks. furthermore, geometry is a topic where we observe the most effectiveness of ∅cot prompting across models. this leads us to argue that for llms, geometry requires creativity (scho-evers et al., 2022)-hallucination-rather than reasoning, to perform well. we also see that counting & probability, which shows performance increase when using null-shot prompting, is another potential topic where reasoning and creativity are both necessary. additionally, we observe moderate performance improvements in pre-algebra and algebra, in contrast to the experiments in section 4, where we observe the improvement more from intermediate algebra. future studies may utilize the previously mentioned interpretability approaches @xcite to further explore how llms pay attention to tokens related to reasoning or hallucination in prompts to better understand this phenomenon. ## ablation studies we perform ablation studies to better understand how placement position of the null-shot phrase and each component in the phrase affects performance. we conduct experiments to assess those aspects in section 7.1 and section 7.2, respectively. in general, we observe that placing the phrase at the end yield the maximum performance, and all components contribute to different degree of improvements, and combining all components, as in our phrase, would be most suitable across tasks. ## conclusions we present various experiments to investigate an intriguing phenomenon when providing llms with a prompt eliciting and exploiting hallucination and observe various performance changes. we observe that null-shot prompting is effective for chat-tuned transformer llms. we also observe that nullshot prompting exhibits its effectiveness for reading comprehension and mathematics-related tasks. given the hallucination detection results, it also reveals a surprising conclusion that null-shot prompting is also effective for increasing llms' abilities for detecting hallucination. combining reasoning and hallucination, ∅cot prompting, shows that some mathematical topics problems require both reasoning and creativity to perform well. we hope this paper serves as an initial step towards a better understanding of how hallucination in prompts affects llms.