Small Encoders Can Rival Large Decoders in Detecting Groundedness (2025)

Abstract: Augmenting large language models (LLMs) with external context significantly improves their performance in natural language processing (NLP) tasks. However, LLMs struggle to answer queries reliably when the provided context lacks information, often resorting to ungrounded speculation or internal knowledge. Groundedness – generating responses strictly supported by the context – is essential for ensuring factual consistency and trustworthiness. This study focuses on detecting whether a given query is grounded in a document provided in context before the costly answer generation by LLMs. Such a detection mechanism can significantly reduce both inference time and resource consumption. We show that lightweight, task-specific encoder models such as RoBERTa and NomicBERT, fine-tuned on curated datasets, can achieve accuracy comparable to state-of-the-art LLMs, such as Llama3 8B and GPT4o, in groundedness detection while reducing inference latency by orders of magnitude.

## introduction large language models have demonstrated remarkable capabilities in various tasks, from text generation @xcite ) to question answering @xcite . however, their tendency to hallucinate when the query lacks support from the provided context or when the model faces noisy retrieval @xcite raises concerns regarding their reliability @xcite . ideally, llms should only answer if the provided context contains enough information to answer the question when combined with the model's parametric knowledge. otherwise, the model should abstain from answering or ask for more information. some works have investigated this issue by evaluating rag-llms in the presence of irrelevant information in the context @xcite @xcite . however, these works lack a consistent definition of "relevant information". it could refer to anything from content that directly answers the question to material that is only loosely associated with the topic. this study defines relevance as information that directly contributes to answering the question and frames the problem as a classification task that distinguishes between relevant and irrelevant information. encoders, such as bert @xcite , roberta @xcite , and nomicbert @xcite , have demonstrated strong capabilities in capturing contextual relationships and producing high-quality embeddings for downstream tasks, which can be useful for factual verification. on the other hand, decoders such as llama have demonstrated strong capabilities in open-ended text generation. however, they often require significantly more computational resources as shown in our analysis reveals two key findings: our findings suggest that practitioners can utilize encoder models for groundedness detection in their workflows, achieving similar efficacy to llms while significantly decreasing computing costs. ## related work a recurring issue in generative models is hallucinations, where the model produces information that appears plausible but is not grounded in reality. @xcite provide a comprehensive taxonomy of hallucinations, categorizing them into intrinsic when arising from model biases, and extrinsic when resulting from data limitations. the former occurs when the generated text includes factual inaccuracies or entities not present in the input, while the latter occurs when the model generates content not supported by the available data or context. groundedness in nlp refers to a model's ability to generate outputs consistent with factual knowledge and the given input context. retrievalaugmented generation (rag) @xcite improves groundedness by incorporating external sources during inference. however, given that rag leads to very long contexts, the "lost in the middle" problem @xcite appears, where models struggle to use information positioned in the middle of long input sequences. some works have tried to solve the problem by focusing on improving the generation and retrieval quality, often by fine-tuning one or more components @xcite . however, those works consider relevant documents only for their analysis. as rag systems surface top-ranked documents, they can still include irrelevant distractions @xcite @xcite . several methods employ a model to predict relevance scores within a larger pipeline @xcite @xcite but none of them tried encoders for relevance detection. ## experimental setup our experiments aim to evaluate the ability of encoder-based models to determine whether a given query is grounded in the provided context before engaging in the computationally expensive process of answer generation by large language models. we work with a dataset d, consisting of instances @xmath0, c), where q represents the query, c represents the provided context, which may or may not contain sufficient information to answer q. the objective is to train a model that can classify each (q, c) pair into relevant or irrelevant, based on whether the context provides enough support for answering the query. during training and inference, we don't answer the question, as our goal is to detect irrelevant context before answering to avoid unnecessary computation. in other words, our approach complements existing retrieval mechanisms rather than substituting them. specifically, our encoder-based groundedness detection step assumes that contexts have already been retrieved and serves as a subsequent filtering mechanism. this step aims to prevent unnecessary computational overhead by ensuring that expensive inference by large language models (llms) is only invoked when contexts are sufficiently grounded. ## future work future research should further explore how groundedness detection methods can handle scenarios involving multiple documents, where collectively supportive information must be integrated to accurately assess groundedness. investigating mechanisms for aggregating passage-level judgments could improve multi-hop question answering, a scenario not specifically addressed in this study. additionally, addressing the detection of internal contradictions within retrieved contexts, a limitation of the current method, represents an important avenue for future research. incorporating factual consistency checks could significantly enhance the robustness and reliability of groundedness detection systems. ## conclusion this paper investigates the task of predicting whether a question is supported by the document provided in context before the computationally expensive answer generation. our experiments reveal that lightweight encoders such as roberta and nomicbert perform well and can closely match the performance of state-of-the-art llms such as llama-8b in some cases. these findings highlight the potential of encoders as an efficient approach to improving the groundedness of nlp systems.