Preemptive Answer “Attacks” on Chain-of-Thought Reasoning (2024)

Abstract: Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. However, the robustness of this approach warrants further investigation. In this paper, we introduce a novel scenario termed preemptive answers, where the LLM obtains an answer before engaging in reasoning. This situation can arise inadvertently or induced by malicious users by prompt injection attacks. Experiments reveal that preemptive answers significantly impair the model’s reasoning capability across various CoT methods and a broad spectrum of datasets. To bolster the robustness of reasoning, we propose two measures aimed at mitigating this issue to some extent.

## methods in this section, we first introduce the method for simulating preemptive answers, followed by a discussion of possible mitigation strategies. ## conclusion this paper investigates how preemptive answers affect llm's reasoning capability within chainof-thought (cot) contexts, showing that such answers can reduce reasoning performance. subsequently, we introduce two mitigation strategies that, although beneficial, do not fully resolve the issue. our findings spotlight a new dimension of llm robustness and pave the way for future work on enhancing reasoning resilience.