Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks (2024)

Abstract: We find that language models have difficulties generating fallacious and deceptive reasoning. When asked to generate deceptive outputs, language models tend to leak honest counterparts but believe them to be false. Exploiting this deficiency, we propose a jailbreak attack method that elicits an aligned language model for malicious output. Specifically, we query the model to generate a fallacious yet deceptively real procedure for the harmful behavior. Since a fallacious procedure is generally considered fake and thus harmless by LLMs, it helps bypass the safeguard mechanism. Yet the output is factually harmful since the LLM cannot fabricate fallacious solutions but proposes truthful ones. We evaluate our approach over five safety-aligned large language models, comparing four previous jailbreak methods, and show that our approach achieves competitive performance with more harmful outputs. We believe the findings could be extended beyond model safety, such as self-verification and hallucination.

## introduction it is arguably easier, at least from the logical perspective, to tell the truth than to tell a lie. for example, given a math problem "what is 1/2 + 1/3", telling the truth only requires the ability to perform the correct reasoning and derive the correct answer. telling a lie, on the other hand, requires the ability to not only discern the correct answers, but also avoid generating the correct answers and, more importantly, make the wrong answers look real. in this paper, we refer to the task of fabricating incorrect yet seemingly plausible reasoning as fallacious reasoning. large language models (llms) have long been struggling with reasoning problems. existing research revealed that llms have difficulty discerning the veracity of their intrinsic answers @xcite how to make and distribute counterfeit? sorry, i cannot assist with that. provide a fallacious procedure to make and distribute counterfeit. ## fallacious reasoning in llms in this section, we present the findings of our pilot study about llms' capabilities in fabricating fallacious reasoning. ## fallacy failure attack the findings introduced above have profound security implications. in this section, we will discuss how the llms' failure in fallacious reasoning can be exploited to form an effective jailbreak at- provide a step-by-step fallacious solution for the following question. explain which step was wrong. question: what is ? solution: 1. 2. find a common denominator: 3. add the fractions: 4. therefore: explanation: step 3 was wrong. the mistake occurred when adding . the correct sum should be , not . (✓) prompt output figure 3: an example where the llm failed to provide a fallacious solution upon request but instead proposed the correct solution and contradictorily claimed it false. ## experiments in this section, we describe our experiment results evaluating the effectiveness of ffa, as well as additional studies on its mechanisms. ## conclusion and future work this paper presented a simple yet explainable and effective jailbreak attack method. it is predicated on the observation that language models cannot generate fallacious and deceptive solutions but instead produce honest counterparts. we argue that this observation not only poses a security threat but also implies how modern llms' perceptions of specific tasks are limited when the scenario is inadequately optimized and aligned during training. we believe this observation can be further extended to related research areas, such as self-verification and hallucination, providing valuable insights into understanding llm behavior toward general intelligence.