Towards the Roots of the Negation Problem: A Multilingual NLI Dataset and Model Scaling Analysis (2025)

Abstract: Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language models (LLMs) and remain underexplored.We constructed and published two new textual entailment datasets NoFEVER-ML and NoSNLI-ML in four languages (English, Czech, German, and Ukrainian) with \textit{paired} examples differing in negation. It allows investigation of the root causes of the negation problem and its exemplification: how popular LLM model properties and language impact their inability to handle negation correctly.Contrary to previous work, we show that increasing the model size may improve the models’ ability to handle negations. Furthermore, we find that both the models’ reasoning accuracy and robustness to negation are language-dependent and that the length and explicitness of the premise have an impact on robustness. We observe higher accuracy in languages with relatively fixed word order like English, compared to those with greater flexibility like Czech and German.Our entailment datasets pave the way to further research for explanation and exemplification of the negation problem, minimization of LLM hallucinations, and improvement of LLM reasoning in multilingual settings.

## introduction understanding negation is a cornerstone of sentiment analysis @xcite , logical reasoning and nuanced communication. still, it often remains a 'pink elephant in the room' for large language models (llms). despite their impressive advancements, effectively understanding and processing negation poses a fundamental, and often underestimated, challenge to their reliability and logical reasoning capabilities. processing negation is critical for reliable performance, and leads to hallucinations as @xcite showed on four tasks with negation: 'false premise completion', 'constrained fact generation', 'multiple choice question answering', and 'fact generation'. vision-language models do not understand negation either @xcite : when asked to provide a picture with no elephant in the room, they put an elephant there anyway. it appears that negation tokens (e.g., "not") have a limited effect on the representations learned distributionally @xcite . this difficulty, sometimes termed negation blindness or no syndrome, is well-documented. studies show llms may struggle to distinguish between facts and their negations, misunderstand the semantic impact of negative particles, and fail to generalize negation handling robustly, even with instruction tuning. ## datasets we evaluate the accuracy of the models in our experiments using two datasets that have been originally formatted for the natural language inference (nli) task, i.e., containing premise-hypothesis pairs and the label denoting their logical relationship @xcite . @xcite is a popular nli dataset focusing on the evaluation of general and common sense knowledge of the llms. the premises are captions of images capturing everyday situations and, as such, are quite short, consisting of one to two sentences. the hypotheses have been generated manually by human annotators. fever-nli @xcite is a modification of the fever dataset, which originally focused on the task of fact extraction and verification. the dataset consists primarily of factual statements retrieved from wikipedia. both of the original datasets are available under permissive creative commons licenses, the snli dataset is licensed under cc by-sa 4.0 and the fever nli dataset is licensed under cc by-sa 3.0. as we aim to study the effects of negation on the accuracy of the models, we modify the format of these datasets for the te task and extend them to contain the negation of the original hypothesis. the original datasets contain a low percentage of hypotheses with common negation markers, such as no, n't, not, nobody, neither, or nothing, and differ in the verbosity of premises, as seen in the ## methodology to study the effect of negation on the reasoning reliability of large language models, we evaluate them on textual entailment @xcite . in this task, the model receives two claims, a premise and a hypothesis, and decides whether the hypothesis follows from the premise. in our experiments, we inspect whether the presence of lexical negation (e.g. "not") in the hypothesis causes degradation in the accuracy of the language model reasoning. for this purpose, we prepare a set of paired evaluation sentences that differ only in a negation of the primary verb. we call the variants hypotheses without negation and hypotheses with negation based on whether they contain a negation. necessarily, if one follows from the premise, the other contradicts it, and vice versa. if the models' reasoning is robust to negations, its accuracy should be the same between the two groups. we deliberately chose such a simple task in order to minimize confounding factors and to demonstrate that the difficulty with negation arises even in the most basic form of the task. all of the evaluated models are publicly available state-of-the-art models based on the transformer architecture @xcite , have open weights, a multi-lingual pre-training, and are fine-tuned to follow user instructions. in the evaluation, we prompt the model to judge if a hypothesis follows from a premise or not. we let the model finish a response "the answer is:" with either true or false using a greedy decoding. to ensure the response follows the predefined structure, we employ outlines (willard and louf, 2023) and lmfe @xcite . more details of the prompt format can be found in appendix b. our implementation for running the experiments is publicly available as a github repository @xcite . ## results we summarize our results in this section, including the results of the statistical tests to support these claims. we find that accuracy and negation sensitivity improve with increasing model size and the information specificity of dataset premises, while variations in languages are not statistically significant. unless otherwise stated, for the statistical tests, we are using an alpha level (significance threshold) α = 0.05. more details of the specific values used during computation can be found in appendix c. model size has a positive correlation with models' ability to handle negation. figure 2 shows the accuracy change of different models for the hypotheses with and without negation. we found that all evaluated models perform better than a random guess and that the larger models narrow the gap between the accuracies, with two outliers on the snli dataset: a slight increase on the middlesized llama model (8b) and a marked drop on the middle-sized mistral model (22b). using the results shown in figure 4 , we compute the correlation of the model size and the relative accuracy change for each evaluated dataset, as well as the overall correlation. using the shapiro-wilk test, we find that the majority of datasets do not have a normal distribution over the different model sizes. thus, we compute correlation using the spearman correlation coefficient, as the pearson correlation coefficient presumes a normal distribution of values. the results indicate a strong positive correlation between model size and the relative accuracy change on the fever dataset and a weak-tomoderate positive correlation on the snli dataset. for an averaged relative accuracy change for all languages per model, the results indicate a strong positive correlation with the spearman correlation coefficient of 0.867, supporting out rq1. we posit that the differences in correlation are due to different philosophies behind the datasets. the fever dataset was originally based on the fact verification dataset and consists of factually specific premises that are sufficient for the prediction of the textual entailment. the snli dataset, on the other hand, relies more heavily on implicit semantical patterns that the models learn during training, introducing an uncertainty we cannot easily predict. our findings are in contrast to the results of (truong et al., 2023, section 3, finding 1), where it is posited that pre-trained language models perform comparably or worse than a random baseline, and model scaling has almost no effect. the difference in our findings can be caused by a particular choice of the evaluated models and their level of fine-tuning, or the increase in the quality of the models themselves since the prior study. differences in methodology may also contribute, as our benchmark specifically focuses on simple examples of lexical negation. another plausible explanation might be that the measured accuracy strongly benefits from fine-tuning of prompts. larger lms are more sensitive to negation for te task. similarly to the relative accuracy change, we compute the correlation between model size and the model's sensitivity to negation. as per the computed p-values for the shapiro-wilk test, we can see that the majority of our negation sensitivity metric across datasets does not have a normal distribution either. therefore, we are again using the spearman correlation. the results indicate a very strong positive correlation of negation sensitivity with the model size on the fever datasets, with a rather weak correlation on the snli datasets. for an average negation sensitivity for all languages per model, the results indicate a strong positive correlation, with a spearman correlation coefficient of 0.866. larger models are less likely to classify pair hypotheses with the same textual entailment label, and the impact of negation is more effectively propagated through the model. one outlier in our results q w e n ( 1 . 5 b ) l l a m a ( 3 b ) q w e n ( 7 b ) l l a m a ( 8 b ) m i s t r a l ( 1 2 b ) m i s t r a l ( 2 2 b ) l l a m a ( 7 0 b ) q w e n ( 7 2 b ) m i s t r a l ( 1 2 3 b ) model (size) 0.5 0.6 0.7 0.8 0.9 1.0 accuracy -28.0% language plays a role in the models' accuracy. figure 3 shows the accuracy change on different models with regard to the language of the dataset. as the language is a categorical variable, instead of correlation, we compute and compare the variance of the relative accuracy change across the four evaluated languages (ces, eng, deu, ukr), excluding the czech manually translated dataset. we aggregate the values shown in figure 4 per language and use the friedman test as our analysis method. our null hypothesis is that all languages have the same average of relative accuracy change, and the alternate hypothesis is that the averages are not the same. analysis has shown that for our degrees of freedom d @xmath0, d @xmath1, the results are f -@xmath2.000 and p-@xmath3.0046. as the p-value is less than our alpha level, we can reject our null hypothesis. thus, we conclude that there is a significant difference in the relative accuracy change across different languages. delving further and applying the wilcoxon signed-rank test on the results of language pairs per specific dataset, we find that there is a statistically significant difference on the czech snli dataset and all of the other languages, with the models performing worse on the czech dataset. on the fever dataset, the models performed the worst ces fever manual ces fever ces snli eng fever eng snli deu fever deu snli ukr fever ukr snli dataset llama (3b) llama (8b) llama (70b) qwen (1.5b) qwen (7b) qwen (72b) mistral (12b) mistral (22b) mistral (123b) model (size) -15.9 % -12.4 % -15.5 % -12.2 % -7.6 % -9.0 % -6.2 % -13.1 % -5.6 % -11.0 % -8.7 % -14.0 % -7.4 % -8.6 % -8.2 % -12.6 % -8.8 % -9.0 % -1.9 % -1.2 % -10.2 % -1.4 % -5.1 % -1.7 % -8.6 % -2.2 % -8.2 % -29.0 % -28.2 % -37.4 % -28.0 % -33.8 % -29.5 % -38.6 % -25.4 % -32.8 % -9.8 % -7.4 % -17.1 % -5.3 % -16.9 % -6.8 % -17.4 % -7.9 % -10.7 % -5.8 % -3.9 % -17.7 % -2.4 % -14.1 % -3.8 % -14.0 % -4.5 % -14.8 % -10.9 % -10.1 % -16.0 % -8.6 % -5.2 % -8.0 % -14.1 % -9.3 % -12.4 % -5.4 % -4.4 % -19.5 % -3.6 % -24.3 % -4.3 % -18.6 % -4.2 % -12.0 % -2.7 % -1.3 % -13.1 % -1.5 % -11.1 % -2.0 % -10. our results partially conform to our presuppositions in rq2 that the models will perform worse in languages with higher word-order flexibility. how-ever, part of the observed differences may also be explained by language-specific phenomena, such as negative concord in czech, where a single negation is expressed by multiple negated words, or by biases in the models, as english and german are likely more prevalent in their training data. ## conclusions and future work this work presents novel datasets enabling the evaluation of language models' robustness in handling negations in four languages.we have extended existing fever @xcite and @xcite datasets to include paired hypotheses that are logically opposite to each other by negating the primary verb in the original hypothesis. we have evaluated the models' accuracy separately on hypotheses with and without negation and computed the accuracy drop that occurred due to the addition of negation in the hypothesis. new datasets nofever-ml and nosnli-ml are freely available @xcite , together with the code used in the negation project @xcite . we find that model size plays a key role in handling negation correctly, countering the previous work of @xcite . the results on differing language datasets only partially match the initial expectations, with models performing worse on the czech configuration. this may be due to the relative simplicity of hypotheses, which do not lend to the inclusion of more complex negation structures, such as the negation token being farther away from the verb, as may be the case in german. in regard to the differing accuracy from one dataset to another, we hypothesize that the accuracy is also heavily dependent on the specificity of information in the premise; the more common-sense snli dataset performed worse than the more factexplicit fever dataset. our finding that negation handling is less reliable on common-sense tasks (snli) suggests a potential source for hallucinations, where models may default to learned semantic patterns over strict logical constraints. improving robustness to negation is, therefore, a direct step towards more factually grounded and reliable llms. our work provides new resources for addressing the deficiency of language models in handling negations. it shows that while the 'pink elephant' of negation is still in the room, larger models are becoming better at seeing what isn't there, particularly when given a clear description of the room's other contents. in future, studying and understanding which part of model architecture is responsible for the problems with negation is our ultimate goal. we will use the lens of llm-microscope @xcite to look at the tokens that play decisive role, depending on subsets of our newly created datasets. this project has received funding from the european union's horizon europe research and innovation programme under grant agreement no. 101070350 and from uk research and innovation (ukri) under the uk government's horizon europe funding guarantee (grant number 10052546). the contents of this publication are the sole responsibility of its authors and do not necessarily reflect the opinion of the european union.