N ev IR : Negation in Neural Information Retrieval (2024)

Abstract: Negation is a common everyday phenomena and has been a consistent area of weakness for language models (LMs). Although the Information Retrieval (IR) community has adopted LMs as the backbone of modern IR architectures, there has been little to no research in understanding how negation impacts neural IR. We therefore construct a straightforward benchmark on this theme: asking IR models to rank two documents that differ only by negation. We show that the results vary widely according to the type of IR architecture: cross-encoders perform best, followed by late-interaction models, and in last place are bi-encoder and sparse neural architectures. We find that most current information retrieval models do not consider negation, performing similarly or worse than randomly ranking. We show that although the obvious approach of continued fine-tuning on a dataset of contrastive documents containing negations increases performance (as does model size), there is still a large gap between machine and human performance.

## introduction recent work in natural language processing (nlp) has shown that language models (lms) struggle to understand text containing negations @xcite and have poor performance compared to humans. this unresolved problem has downstream implications for information retrieval (ir) models, which use lms as the starting backbone of their architectures. however, work on negation in ir has mainly focused on pre-neural (e.g. no lm) retrieval @xcite @xcite , with no research into how negation affects modern neural ir. this failure to understand negation in ir can lead to devastating consequences in high stakes 1 code and data are available at https://github.com/ orionw/nevir had a seizure now what? hold the person down or try to stop their movements. put something in the person's mouth (this can cause tooth or jaw injuries) administer cpr or other mouthto-mouth breathing during the seizure. give the person food or water until they are alert again. figure 1 : negation is something not well understood by ir systems. this screenshot shows google search making a deadly recommendation because of its failure to catch the negation in the article (e.g. "do not ..."). we aim to fill this gap in the literature by providing a benchmark for negation evaluation in information retrieval, dubbed nevir (pronounced "never"). nevir builds off of existing work in negation @xcite by using 2,556 instances of contrastive document pairs that differ only with respect to a crucial negation. we then crowdsource query annotations for the two documents in each pair, where each query is only relevant to one of the respective documents and is irrelevant to the other document (figure 2 ). by doing so, we can test whether models correctly rank the documents when accounting for the negation. we find that nearly all ir systems ignore the negation, generally scoring one document of the because it is resistant to corrosion, nickel was occasionally used as a substitute for decorative silver. nickel was also occasionally used in some countries after 1859 as a cheap coinage metal (see above), but in the later years of the 20th century, it was replaced by cheaper stainless steel (i.e. iron) alloys, except in the united states and canada. because it is resistant to corrosion, nickel was occasionally used as a substitute for decorative silver. nickel was also occasionally used in some countries after 1859 as a cheap coinage metal (see above), but in the later years of the 20th century, it was replaced by cheaper stainless steel (i.e. iron) alloys, throughout the united states, canada, and elsewhere in the americas. ## creating nevir we test negation in neural ir using a contrastive evaluation framework, which has shown great utility in understanding neural models (section 2.5). ## fine-tuning on nevir we use the top performing model from nonsparse categories: multi-qa-mpnet-base-dot-v1 from sentencetransformers, colbertv1, and monot5-base from pygaggle. we fine-tune them using sentencetransformers, the original col-bertv1 code, and the original pygaggle code. we train for 20 epochs and evaluate them on nevir test and msmarco dev after each epoch. figure 5 shows that fine-tuning on negation data improves performance significantly, but still leaves a large gap to perfect (and the human score of) 100% paired accuracy. as would be expected, the large monot5 model quickly learns and then overfits to the data (while quickly losing perfor- ## discussion and implications implication for current systems ir model's performance on nevir indicates that first stage retrievers do not take negation into account when doing retrieval. thus, to perform well on negation with current models, expensive cross-encoder rerankers are necessary but not sufficient to achieve good results. furthermore, our analysis indicates that in order to best learn negation (and significantly improve their performance), models should incorporate negation into their training data. thus, when high precision for negation retrieval is not needed (e.g. some first stage retrieval settings), current models may be effective, as they will retrieve lexically similar documents regardless of negation. however, in order to have high-precision retrieval with negation (and documents with both negation and non-negation have high lexical over-lap), expensive cross-encoders are the only current models that perform better than random ranking. nevir provides the only dataset for measuring and improving retrieval with negation. implications for current users anecdotally, most users tend to avoid using negation queries in production ir systems like google search. this may be a self-reinforcing problem, as users have found poor results when they use negation in search and hence avoid using negations in the future. for example, the webpage for the university of utah article that is shown in figure 1 has since been updated and currently includes no negation words. thus, it is unclear whether queries with negation are less common because of people's actual information needs or because production systems have biased users (and content creators) into an avoidance of negation. we hope that by introducing a benchmark for ir evaluation we can help enable these types of queries in the future. ## conclusion we proposed to benchmark negation in neural information retrieval and built a benchmark called nevir to explore this problem, crowdsourcing annotations from mechanical turk. we found that modern ir models perform poorly on this task, with cross-encoder models performing the best (slightly above random performance) and all other architectures (bi-encoder, sparse, and late-interaction) performing worse than random. further we showed that simply including negation in fine-tuning provides significant gains, although there is still room for improvement to reach human performance. we hope that this benchmark inspires future work into improving information retrieval model's ability to recognize negation.