This is not a Disimprovement: Improving Negation Reasoning in Large Language Models via Prompt Engineering (2025)

Abstract: Negation reasoning remains a challenge for large language models (LLMs), often causing incorrect interpretations of negated statements. In this study, we analyze various LLMs for their handling of negation and propose two genres of prompts (*Warning-based* and *Persona-based*), which improve overall absolute accuracy by up to 3.17% and distractor negation accuracy by up to 25.14% over most competitive baselines. Next, we assess the robustness of LLMs by reordering prompts while preserving meaning, observing instability linked to positional encoding schemes. Further, we introduce a negative token attention score (NTAS) to quantify attention to negation words. From the comprehensive analysis, we point out that within a specific LLM family, the performance of a model (measured using accuracy) correlates more with NTAS than with model size. The code is publicly available: [https://github.com/Joshua-Dias-Barreto/This-is-not-a-Disimprovement](https://github.com/Joshua-Dias-Barreto/This-is-not-a-Disimprovement)

## introduction large language models (llms) have achieved state-of-the-art performance across diverse natural language processing (nlp) tasks, indicating their grasp of complex syntactic and semantic patterns. however, llms often struggle to correctly interpret sentences where negation alters the truth conditions of a proposition @xcite @xcite @xcite . various methods have been proposed to improve neural models' robustness to negated inputs. @xcite improve the bert understanding of negation by combining unlikelihood training with syntactic data augmentation using negated lama @xcite . in another study, @xcite develop a pretraining strategy specifically designed to address negation, while @xcite explore paraphrasing negations into affirmative terms. studies have also been attempted to investigate the robustness of llms with variations of prompts, size of models, etc. for example, llms have been found to be sensitive to variations in prompt format @xcite and content @xcite . on the other hand, @xcite highlight that even minor prompt perturbations can significantly degrade performance, especially for negation. in one of the recent works, @xcite observe that llms exhibit a negative bias and propose a negative attention score (nas) to measure it. even though there have been studies dealing with negation reasoning, very few attempts have been made to improve the performance of llms for the same. hence, in this paper, we propose a prompt engineering-based framework to improve the performance of llms for negation reasoning. towards that objective, we pose negation reasoning as a binary classification task where an llm is supposed to classify a given statement as 'true' or 'false' (figure 1 ). in our study, such statements are taken from a gold standard dataset, namely thisis-not-a-dataset proposed by @xcite , and as the baseline prompt, we consider a simple question -'is the following statement true or false?' @xcite . we propose two different genres of prompts-warning-based and persona-based to improve the negation reasoning capabilities and experiment with five different llm families @xcite , gpt-2 @xcite , qwen1.5 @xcite , qwen2.5 @xcite and ibm granite 3.0 @xcite ). note that the baseline prompt is appended with the proposed prompts to make the entire prompt. we observe a distractor negation accuracy improvement of 25.14% and 10.9% for ibm-granite-3.0-2b using warning-based prompt and persona-based prompts, respectively ( next, we introduce a negative token attention score (ntas) to quantify attention to negative words (e.g., "no", "never", and "not") and examine its link to improved negation reasoning performance. further, we assess llm robustness by swapping prompt order (proposed+baseline vs. baseline+proposed) and grouping models by positional encoding: relative, absolute, and rotary. our analysis shows that models with relative encoding (e.g., flan-t5) show stable performance, while others vary when prompts are reordered. we also find that larger models don't always outperform smaller ones in negation tasks. experiments show that within a specific llm family, the performance of a model (measured using accuracy) correlates more with ntas than with model size. task formulation: we follow the task formulation proposed by garcía-ferrero et al. (2023), which focuses on the binary classification ('true' or 'false') of sentences in the dataset. given a statement st, the model must generate true or false tokens. following the work of garcía-ferrero et al. (2023) and scheurer et al. (2023), the answer a is computed as follows: @xmath0.5 false otherwise dataset: the dataset introduced by garcía-ferrero et al. ( 2023 ) comprises approximately 400,000 semi-automatically generated english sentences encapsulating commonsense knowledge, with about two-thirds containing various forms of negation. the dataset features sentences structured into 11 distinct patterns, each designed to test different aspects of negation understanding. these patterns are based on wordnet @xcite relations, including synonymy, hypernymy, antonymy, and meronymy, as well as the semantic roles like agent, instrument, etc., which are defined by morphosemantic links @xcite . we use its test set, which has approximately 90,000 sentences. ## proposed framework llms: we use a total of five llm families, which are again categorized into three classes based on types of positional encodings. note that positional encodings indicate how llms handle sequential information. we only use models available through open-access apis.the details of the llm families, along with the classes, are described below. relative positional encoding @xcite captures the distance between tokens to focus on relative positions, which is used by flan-t5; absolute positional encoding @xcite assigns fixed positional values based on token order, which is used by the gpt-2 family; rotary positional encoding @xcite applies a rotational transformation for flexible relative positioning, which is used by the qwen1.5, qwen2.5, and ibm granite 3.0 families foot_0 . next, we describe the two genres of prompts to improve negation reasoning in llms and negative token attention score (ntas) to hypothesize why certain prompts or models have higher accuracies. ## experimental results and analysis experimental set-up: for all experiments, we used a single nvidia quadro rtx 6000 ada generation gpu. to ensure uniform inference behaviour, flash attention @xcite is disabled for all models. we use 'bfloat16' precision where supported, which helps optimize memory usage @xcite . to facilitate efficient inference on a single gpu, we employ 4-bit quantization @xcite . we mainly focus on accuracy as the evaluation metric. ## conclusion this study aimed to enhance llms' negation reasoning through two genres of prompts: warningbased and persona-based. experiments showed that small prompt variations can significantly boost performance up to 3.17% over baseline accuracy and 25.14% for distractor negation accuracy. we proposed a negative token attention score (ntas) to quantify attention to negative words. our analysis showed that inconsistencies with prompt reordering are likely tied to positional encodings. though larger models usually perform better, this trend is inconsistent; accuracy correlated better with model ntas than size within the same family of llms. immediate future work could be to do a thorough pattern-wise analysis of the eleven linguistic patterns present in the target dataset to identify which patterns llms handle well and which they struggle with. using other binary labels such as yes/ no instead of true/ false is something that can be explored. examining multilingual nega-tion reasoning to evaluate generalization across languages and investigating whether effective prompts in one language transfer to others could be another future direction to explore.