Benchmarking Query-Conditioned Natural Language Inference (2025)

Abstract: The growing excitement around the ability of large language models (LLMs) to tackle various tasks has been tempered by their propensity for generating unsubstantiated information (hallucination) and by their inability to effectively handle inconsistent inputs. To detect such issues, we propose the novel task of Query-Conditioned Natural Language Inference (QC-NLI), where the goal is to determine the semantic relationship (e.g. entailment or not entailment) between two documents conditioned on a query; we demonstrate that many common tasks regarding inconsistency detection can be formulated as QC-NLI problems. We focus on three applications in particular: fact verification, intrinsic hallucination detection, and document inconsistency detection. We convert existing datasets for these tasks into the QC-NLI format, and manual annotation confirms their high quality. Finally, we employ zero- and few-shot prompting methods to solve the QC-NLI prediction problem for each task, showing the critical importance of conditioning on the query.

## introduction natural language inference (nli) has become a standard method to detect inconsistencies across pairs of sentences or documents @xcite @xcite . the need to detect inconsistencies has become even more pronounced in the era of large language models (llm's) -much work has shown that such models produce internally contradictory text, information inconsistent with external knowledge, or information that contradicts input evidence, as in a retrieval-augmented generation (rag) system @xcite . to detect these types of shortcomings, we propose the task of query-conditioned nli and show how various inconsistency detection tasks can be solved using it. query-conditioned nli aims to determine the relationship between the premise and hypothesis given a query. the inclusion of the query distinguishes our task from standard (nonconditional) nli tasks, which only consider the relationship of the entire premise to the entire hypothesis; these paradigms are illustrated in figure 1 . consider the following example from traditional nli: premise: jack is working in the park. hypothesis: jack is sleeping in the park. label: contradiction a typical nli dataset would consider these sentences a contradiction because jack can't be sleeping if he is working. however, in conjunction with the query "where is jack?", the label should be entailment because both sentences state that jack is in the park. hence, the query defines the aspect of the input sentences for which the nli label should apply. this problem is even more pronounced when the premise and hypothesis are longer documents, which often contain both consistent and inconsistent information. in this paper, we introduce the task of queryconditional nli and provide a benchmark dataset for it. in particular, we show that the tasks of fact verification, intrinsic hallucination detection, and inconsistent document detection can be cast as a qc-nli problem, and we adapt existing datasets into our framework (examples may be seen in ## related work traditional natural language inference our work builds on the task of natural language inference @xcite , which is traditionally done at the sentence-level: the premise and hypothesis are each single sentences, and the task is to determine whether the premise entails, contradicts, or is neutral to the hypothesis. typical datasets for this task are the recognizing textual entailment challenge tasks @xcite and the stanford natural language inference dataset @xcite . our work differs from these works in two major ways: first, rather than individual sentences, we focus on paragraphs or entire documents. second, our proposed task is query-conditional nli: we are not concerned with the relationship of the entire premise to the entire hypothesis, but rather a particular aspect of them, determined by the query. ## solving the qc-nli benchmark in addition to developing qc-nli datasets for qc-nli for various downstream tasks, we present some prediction baselines: given two documents and a query, how do the documents relate with respect to the query? we use three prompting methods: zero-shot prompting, few-shot prompting, and qa+nli prompting. in all cases, we ask for the llm to explain its reasoning before making a final prediction. ## experimental results because the label distributions of the datasets are often unbalanced (see and true negative rates) to evaluate the predictions. note that 0 ≤ @xmath0, with 0.5 indicating random guessing. ## discussion importance of the query the dramatic drop in performance between methods that condition the prediction on the query and those that do not underscores the importance of the query. notably, the drop is not as significant for robustqa and ragtruth as it is for snli and factscore. for ro-bustqa, each document provides either a "yes" or "no" perspective; thus, the query is not strictly nec-essary to solve the task. for ragtruth, the entire hypothesis document is the output of an llm conditioned on the premise and the queryhence, providing the query to the qc-nli task is not as helpful since the hypothesis already addresses it. however, even for these datasets, conditioning on the query is more helpful than not. on the other hand, snli and factscore exhibit dramatic decreases in performance when removing the query from the prediction task. for snli, this is partly due to dataset construction, where we encourage the documents to describe distinct settings. thus, the extraneous material does not provide much information about the nli label, meaning that the query defines the relevant aspect. unlike snli, factscore actually has multiple queries (and labels) per pair of documents. thus, the query is, by design, critical to determining the label: even a majority voting oracle only achieves a balanced accuracy of 0.75, which is over 4 points lower than the average balanced accuracy on the queryconditioned methods. the prospect of qa+nli methods as mentioned in section 6, few-shot prompting generally outperforms both zero-shot prompting and qa-nli for all datasets and models. however, qa+nli (where the system first answers the question on each document, and then makes an nli prediction based only on the answers) performs quite well for snli and robustqa. section 5 makes the point that this type of system may not work well, depending on how the qa system answers the question; so, why does this work for snli and robustqa? for robustqa, the questions can all be answered with "yes" or "no", and each document gives one of these perspectives (see section 4.1). therefore, a qa system on each of these documents will return an affirmative or negative answer, making it easy to determine the label and explaining the rather small drop in performance between the few-shot and qa+nli systems on this dataset. the snli dataset does not contain yes-no questions, but contains rather literal questions about image descriptions. hence, it is also feasible to first answer the question on each image and then perform nli. ragtruth, on the other hand, contains much more difficult queries. most queries are highly open-ended and involve checking multiple sentences in the hypothesis against the premise. thus, a qa system that does not return all necessary parts of the documents will not have the necessary infor-mation to solve the nli task. thus, the documents themselves are important, and this is an important aspect of our qc-nli formulation. it is less clear why qa+nli is not as effective as the end-to-end methods for factscore, since the questions ask about atomic facts. an analysis of the dataset shows that the queries are typically based on the hypothesis document only (as the goal is to check whether various facts in the hypothesis are entailed by the premise); so a possibility is that the qa result on the premise is not very informative. further, the lower performance of qa+nli compared to end-to-end prompting is consistent with other works that suggest that unified approaches outperform cascaded systems @xcite . this discussion suggests that some of our datasets are easier than others for qc-nli. future systems for the qc-nli benchmark thus should be aware of this, and special focus may be necessary for those datasets that contain multiple sentences or facts that need to be checked across the documents. a long way to go finally, none of the methods achieve a balanced accuracy greater than 82.63% (gemini 1.5 pro on factscore); this is only slightly below factscore's ceiling accuracy of 84.31%, which is the quality of the dataset determined by our annotation study. most models achieve 60-70% on the various datasets, slightly lower than their respective ceiling accuracies. even so, while zero-and few-shot prompting methods work well in many cases, some datasets still appear very hard to solve. robustqa, in particular, has a long way to go: the ceiling is 93.94%, but our best method achieves 68.11%. future work can look at more sophisticated methods for solving these tasks, such as conditional neural architectures @xcite @xcite . ## conclusion & future work in this paper, we have proposed the novel task of query-conditioned natural language inference (qc-nli) and shown how it can be useful for three different applications of interest in the nlp community: fact verification, hallucination detection, and inconsistency detection for rag. we adapt existing datasets to create a high quality benchmark for this task, where the premise and hypothesis are each multi-sentence documents. we then propose several baselines: zero-shot prompting, fewshot prompting, and qa+nli (the composition of question-answering and nli). the results have two primary takeaways: the query is important for solving the task (as evinced by the marked drop in accuracy when the query is removed from the prompts); and (2) that the documents are important to solving the task in the sense that first answering the query on the documents and then performing nli inference is not always sufficient. there are many avenues for future work. perhaps most important is the development of higherperforming methods for qc-nli: in particular, the best method on the robustqa dataset performs about 15 points lower than its ceiling. more varied prompt strategies could improve performance and robustness. further, training (or fine-tuning) models on this task may be important to improving performance, and conditional methods could be critical. another direction would be the use of a qc-nli system in downstream applications: for example, eliciting more holistic answers from rag systems by providing documents that proffer different perspectives with respect to the query. in these ways, qc-nli is a useful benchmark with much potential in the nlp community.