A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers (2025)

Abstract: Understanding and solving complex reasoning tasks is vital for addressing the information needs of a user. Although dense neural models learn contextualised embeddings, they underperform on queries containing negation. To understand this phenomenon, we study negation in traditional neural information retrieval and LLM-based models. We (1) introduce a taxonomy of negation that derives from philosophical, linguistic, and logical definitions; (2) generate two benchmark datasets that can be used to evaluate the performance of neural information retrieval models and to fine-tune models for a more robust performance on negation; and (3) propose a logic-based classification mechanism that can be used to analyze the performance of retrieval models on existing datasets. Our taxonomy produces a balanced data distribution over negation types, providing a better training setup that leads to faster convergence on the NevIR dataset. Moreover, we propose a classification schema that reveals the coverage of negation types in existing datasets, offering insights into the factors that might affect the generalization of fine-tuned models on negation. Our code is publicly available on GitHub, and the datasets are available on HuggingFace.

## introduction a key factor contributing to accurate relevance in neural information retrieval (ir) systems, llmbased re-rankers, and retrieval augmented generation (rag) is acquiring language understanding capabilities through pre-training @xcite . despite their extensive training setups, these models show persistent difficulty in handling negation @xcite , both in spoken and written language @xcite . negation is linguistically a complex phenomenon that, while guaranteed to be present in the training regime of any model, takes different forms depending on the task at hand. human comprehension of negation comes as a result of understanding linguistic, morphological, and syntactic construction along with verbal cues (as defined in appendix a.1) and facial expressions @xcite . however, this multifaceted linguistic phenomenon is often reduced to a binary description in language processing systems: does negation exist or not in a specific data set @xcite , and is it encoded or not by a model @xcite . addressing these discrepancies between human and system understanding of negation, we ask the following research questions: (rq1) can we design a comprehensive taxonomy for negation? (rq2) how can this taxonomy be applied to generate a more complete and balanced dataset? (rq3) in what manner does model performance differ when fine-tuned on the taxonomydriven dataset versus prior existing datasets? (rq4) how can this taxonomy be used to understand why models underperform on existing negation datasets? rq1 aims to bring together research from the linguistic literature in a taxonomy on negation. we design our taxonomy to be exhaustive, with no overlap, and relevant to ir tasks. to address rq2, we propose two synthetically generated datasets that cover all proposed negation types. figure 1 illustrates the task alongside the data type represented in our datasets. rq3 analyzes the performance of neural ir models, providing insight into the gap between human understanding and llm encoding of negation. rq4 connects the taxonomy to formalizations that can be used as data classification mechanisms, allowing to study existing datasets and identify reasons why fine-tuning does not guarantee a performance boost. chlorophyll in green plants absorbs light from the blue and red spectrums of light. these absorbed wavelengths drive the photosynthetic reactions by energizing electrons in the chlorophyll molecules. ## motivation negation has a long history in (computational) linguistics. the study of opposition and its expression in the form of negation is a phenomenon that has been debated by, and provoked interest from linguists, logicians, metaphysicians, and philosophers @xcite @xcite . it is a highly complex expression of thought given its apparent simple form @xcite . other challenges are imposed by the ambiguity of the negation scope @xcite , and pragmatic inferences in conversational settings @xcite . proper treatment of negation is essential. understanding negation is vital for retrieval models to provide the correct information to the user. moreover, handling negation is vital to ensure that the retrieved generations are a correct response to the user query, since generated answers are particularly difficult to verify, as they cannot be grounded in established evidence @xcite . equally important is ensuring that rag systems respect user-specified negation and avoid retrieving information the user explicitly does not search for. fine-tuning on negation datasets. one could argue that this problem can be mitigated through fine-tuning @xcite . however, catastrophic forgetting occurs when a model is fine-tuned on a new dataset @xcite , even if its distribution is similar to the original training data. in certain cases, fine-tuning can lead to a degradation of performance in the original training set @xcite . model sensitivity to parameter adjustments is particularly noticeable in information retrieval settings. this has been observed in traditional bert-based architectures @xcite and llms @xcite . although this behavior can be mitigated by freezing the model parameters and adding a language model head that is fine-tuned on a new dataset @xcite , this method restricts the capabilities the model can learn. @xcite shows that fine-tuning on their proposed dataset (nevir) leads to a noticeable decline in msmarco generalization performance. representations of negation. another explanation for models under-performing on negation is a lack of specialized pre-training setups, which can arise from an under-representation of negation in popular pre-training datasets (hossain et al., 2020), although llms have been trained on extensive crawled datasets and exhibit similar shortcomings. an improper training can also be caused by the training objective. while contrastive loss pushes different content to be distant in the representation space, two negated statements are close in content while conveying opposite information. @xcite address the problem of having a misalignment between the training objective and the semantics of negation by proposing an 'unlikelihood' loss function used to further pre-train bert on factually incorrect statements with negation cues. ## related work negation in ir. negation has been studied since early language models, e.g., jumelet and hupkes (2018) investigate the capabilities of lstms to locate the scope of negation, which they evaluate using a parse tree. early work typically examines negation at the atomic sentence level. in contrast, negation in ir must be handled across pairs of queries and documents, as the presence of negation in a query can completely reverse the relevance of a document that otherwise is a semantic match. therefore, ir systems must assess whether both the query and the document share the same polarity. i.e., positive or negative @xcite . negation in ir often takes the form of exclusion, which involves filtering information, and rejection of suggestions, which involves dismissing information, as mentioned by @xcite . having distinct types of negation poses an added challenge to defining it in an ir context, which can therefore be difficult and ambiguous. negation in different modalities. @xcite propose a benchmark for understanding negation across 18 tasks and modalities spanning image, video, and medical data. their experiments reveal that even with large-scale training, modern vision language models (vlms) struggle with negation, often performing at random. the authors show that fine-tuning on large-scale synthetic datasets can approach a 10% increase in performance. however, that forces the model to overfit on negation instead of making it reason on negation, as shown by achieving a good performance on one dataset but not generalizing on negation out of distribution @xcite . retrieval models and llms for retrieval. information retrieval models evolved from lexical matching to dense retrieval, where the similarity between a query and documents is identified in a latent semantic space. these representations can be learned separately, i.e., with bi-and dual encoders, or together, i.e., with cross encoders. dense models have been shown to outperform classical lexical matching in most scenarios @xcite . in addition, llms are being fine-tuned to serve as the backbone of retrieval and ranking tasks @xcite , bringing a boost in performance through their rich representations. llm-based models used for retrieval are constructed on small-scale models, such as bert @xcite and t5 @xcite , or on larger-scale next token prediction models, such as llama @xcite , mistral @xcite and qwen @xcite . data generation using llms. data generation using llms has gained significant attention @xcite @xcite @xcite , and has been shown to be a viable method to expand the training dataset, improving performance in several tasks such as dialog generation @xcite , reasoning @xcite , negation @xcite and exclusionary retrieval @xcite . existing negation datasets. one of the first forays into negation understanding was in the medical domain, where research focused on automatically indexing clinical reports and discharge summaries @xcite . for example, bio-scope @xcite is a corpus of biomedical text mining that focuses on extracting accurate information on biological relations. today, in the ir literature, we have access to publicly available datasets such as nevir @xcite , ex-cluir @xcite , boolquestions @xcite , quest @xcite , and romqa @xcite . while these datasets contain logical operator annotations, the annotation system largely remains a single binary label for the presence of negation. research gap. how is a taxonomy different from linguistic formalisations of negation in logic? aristotle transferred the study of negation from the domain of ontology to logic and language @xcite . the linguistic formalization of negation in logic defines how negation operates within formal systems @xcite , such as in classical logic, where a proposition p is negated through ¬p in which the truth value is flipped, or within modal and nonmonotonic logic @xcite , where it has more nuanced interpretations. in contrast, a taxonomy for negation would categorize different types and functions of negation in language and reasoning, such as lexical @xcite vs. semantical @xcite negation, metalinguistic @xcite vs. descriptive @xcite , or negation as opposition @xcite vs. absence @xcite . although logic treats negation as a formal operation on truth values, a taxonomy explores its diverse roles in communication, cognition, and interpretation. ## methodology we propose (1) a taxonomy for negation that is used to generate (2) two synthetic datasets that can be used for evaluating the performance of neural information retrieval models and for fine-tuning models to become more robust on negation, and (3) a classification mechanism that splits existing datasets into granular types of negation. ## experimental setup throughout this study, we use the gpt-4o-mini model @xcite to conduct experiments that aim to answer our research questions. more precisely, we evaluate retrieval models to reveal the necessity of our taxonomy-driven synthetic data, evaluate categorized existing datasets to show the usefulness of our logic-driven mechanism, and fine-tune to show that a coverage of negation types can help with generalisation. evaluation of the generation. we assess the quality of the generated datasets with human annotation on 5% of the generations, with two annotators evaluating each instance on: (1) relevance of documents to each query, (2) presence of negation, (3) naturalness, (4) coherence, and (5) consistency of information within the document. the annotation was conducted with labelstudio. 4 we assess the annotations on quantitative and qualitative measures, together with the annotator agreement. appendix a.6 illustrates the questions for the annotators, metrics used, alongside further details for the setup. for both performance and inner annotator agreement, we use metrics such as f1-score, average on ordinal scales, and (weighted) cohen's kappa. tables 5 and 6 report the annotation metrics. the main findings are as follows: datasets. we evaluate on three benchmarks. nevir and excluir are two contrastive benchmarks where each instance comprises of two documents and two queries that only differ by a targeted negation, or exclusion. we also use msmarco @xcite dev partition, which is not specifically designed for contrastive pairs, but is used simply as a complex retrieval benchmark. metrics. the metric used to evaluate the task is pairwise accuracy: for each instance queries q 1 , q 2 and documents d 1 , d 2 , the model independently ranks {d 1 , d 2 }. the prediction is correct only when the system places d 1 above d 2 for q 1 and inverts the order for q 2 . random performance for pairwise accuracy is 25%. fine-tuning. we fine-tune three models: col-bertv1, multi-qa-mpnet-base-dot-v1, and mistral-7b-instruct for 20 epochs on the free generated dataset and evaluate on nevir @xcite test and msmarco @xcite dev data. figure 3 : pairwise accuracy on the free generations dataset. the first result column contains the full dataset; later columns represent one negation type each. models are represented by the rows, where i is a shortcut for instruct. on the right, we assign labels expressing the architecture and training objective of each model: the first position shows the architecture, i.e., sparse, bi-encoder, dual encoder, crossencoder, and transformer; the second position shows the training objective, i.e., retrieval, search, similarity, ranking, natural language inference, and next token prediction. for a close-up, see appendix a.7. ## results our experiments are designed to investigate the following hypotheses: (h1) some negation types are better encoded in the model internal representations than others, (h2) model specifics such as architecture, training objective, size and backbone significantly influence performance on negation, (h3) existing datasets have an uneven representation on negation, (h4) fine-tuning on our synthetically generated dataset will show systematic improvement in the downstream task presented in figure 1 . ## conclusion in this study, we propose a philosophy, logic and linguistic-grounded taxonomy for negation along two synthetic datasets that can be used for evaluating existing neural retrieval, ranking and llm reranker models, and for fine-tuning models to increase their capabilities on negation. through our study, we found that (1) cross-encoders and llm rerankers are better at encoding negation, (2) nevir and excluir have a limited coverage of negation types, and ( 3 ) fine-tuning on our synthetic datasets helps performance in a negation domain. these insights confirm that negation is a complex phenomenon and that a thorough taxonomy brings advantages as a starting point for generating fine-tuning data. the taxonomy-based classification of current datasets, together with model evaluation, shows that having a broad coverage of negation types is vital. our fine-tuning experiments confirm that the synthetic datasets bring a performance boost; however, it also indicates that fine-tuning data might not be the sole factor behind model difficulty with negation. the training objective and architectural backbone play a big role in model performance performance. however, different training objectives are a promising direction for future work. moreover, we propose investigating negation in a retrieval setting with a large corpora. moreover, while generalization drops with fine-tuning, we propose investigating the training objective by applying reinforcement learning on negation with a small subset, similar to r1-search @xcite .