Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi- LLM Collaboration (2024)

Abstract: Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps—missing or outdated information in LLMs—might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on held-out sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our abstention methods pinpoint failure cases in retrieval augmentation and knowledge gaps in multi-hop reasoning.

## introduction llms demonstrate impressive capabilities of encoding real-world knowledge in model parameters and leveraging it to aid knowledge-intensive tasks @xcite @xcite . but when such knowledge is missing or unreliable, they resort to hallucinations @xcite and biases @xcite , while still "speaking with confidence." a growing body of work seeks to expand llm knowledge through retrieval augmentation @xcite @xcite @xcite , search engine integration @xcite , and multi-lm collaboration @xcite . however, llm knowledge gaps might always persist due to the ever-evolving nature of knowledge @xcite @xcite @xcite . consequently, we posit that abstaining from generating low-confidence outputs should be a part of llms' functionality, and ask a crucial research question: how to identify knowledge gaps in llms? developing and evaluating robust mechanisms to address the abstain problem improves llm reliability, reduces hallucinations, and mitigates biases due to model uncertainty. we hypothesize that there are three ways to operationalize the abstain problem: (1) from the data perspective, llms should abstain when knowledge is missing in training data or curated from unreliable sources; (2) from the modeling perspective, llms should abstain when knowledge is not encoded in model parameters and cannot be inferred; specifically, we focus on abstaining in questionanswering (abstainqa): given an llm and a question q, we aim to develop an abstain function f (llm, q) → {true, false} to indicate whether the llms should abstain based on the limitations of their internal knowledge @xcite @xcite . to facilitate a comprehensive understanding of existing approaches that can be employed for llm abstention, we first identify 11 methods from previous works and adapt them to incorporate the abstain functionality. we categorize these baselines into (1) calibration-based, such as employing tempera-ture scaling @xcite to calibrate models and set a confidence score threshold for abstention; (2) training-based, such as employing linear probing on llms' hidden layers to evaluate the veracity of generated texts @xcite ; (3) prompting-based, such as prompting llms to request more information before answering questions @xcite ; and (4) self-consistency based, such as generating multiple chains-of-thought @xcite . among the 11 baselines, a majority and often the stronger approaches would require a held-out set for training and hyperparameter tuning, potentially harming generalization across knowledge domains. in addition, these approaches often rely on a "selfreflection" assumption that a single llm could be employed or adapted to evaluate its own generated texts; however, challenges such as hallucination and confirmation biases cast doubt on this assumption @xcite . we propose multi-llm collaboration-based approaches, enabling robust llm abstention through multi-llm collaboration to reflect on generated text in cooperative or competitive settings while removing the need for held-out sets. for multiple llms to work in cooperation (cooperate), the llm employs other models to provide feedback on the proposed answer and reasoning, and synthesizes the outputs into an overall abstain decision. for a competitive setting (compete), the llm is challenged by other llms with conflicting knowledge, producing an abstain decision based on whether it continues to prefer the proposed answer in the presence of conflicting evidence. we evaluate the baselines and proposed collaboration approaches for abstention with three llms, on four knowledge-intensive qa tasks spanning diverse knowledge domains and reasoning. extensive experiments demonstrate that cooperate and compete are strong abstain mechanisms that outperform all baselines in 9 of the 12 settings across tasks and models, achieving an improvement of up to 19.3% in abstain accuracy. further analysis reveals that the proposed collaboration-based approaches could help identify failure cases of retrieval augmentation, pinpoint knowledge gaps in multi-hop reasoning, and more. our contributions are (1) a critical evaluation and typology of diverse existing methods, repurposed to identify knowledge gaps in llms, as well as (2) two novel, robust multi-llm collaboration methods to detect llm knowledge gaps, cooperate and compete. ## identifying knowledge gaps in llms we propose to repurpose existing approaches to enable abstain functionality in the face of knowledge gaps in llms. we conceptually categorize these approaches into calibration ( §2.1), training ( §2.2), prompting ( §2.3), and self-consistency ( §2.4) methods. empirically observing that these approaches are dependent on external knowledge/data sets and often suffer from instability, we propose a new category based on multi-llm collaboration ( §2.5). ## results we present the performance of abstain mechanisms with three llms on four tasks in cooperate and compete are state-ofthe-art abstain mechanisms. our proposed collaboration-based approaches outperform the strongest baseline in 9 out of the 12 settings (3 models and 4 datasets), achieving an average improvement of 10.1% on reliable accuracy scores. we find that cooperate works better with stronger models such as chatgpt: we hypothesize that the complexities of the multi-step feedback generation warrant a stronger base llm. compete emphasizes reliability and greatly avoids wrong answers evident in the high reliable accuracy scores: we observe that llms stick to the original answers only in the most confident scenarios, making compete ideal for abstention when reliability is paramount. @xmath0, the four types of approaches achieved 0.595, 0.576, 0.553, and 0.475 a-acc scores. while simple prompting-based approaches struggle to elicit self-reflection, our proposed cooperate and compete promote collaborative abstention and greatly improve performance. among all baselines, instruction tuning and selfconsistency threshold stand out as two strong approaches: however, they both rely on a held-out set for training and hyperparameter tuning: we further investigate their generalizability in section 5. abstain capabilities potentially correlate with base llm utility. abstain accuracy (a-acc) is a metric independent of llms' underlying performance on a qa dataset and solely evaluates ab-stention quality. for a-acc, mistral-7b, llama2-70b, and chatgpt achieve 0.524, 0.537, and 0.616 a-acc scores on average across approaches and datasets, which aligns well with their relevant superiority on standard benchmarks. in addition, our proposed collaboration-based approaches also work better with the strongest chatgpt, while also improving the abstention performance of weaker llms over baselines. going beyond generic qa sheds light on the limitations of existing approaches. while most baselines could function properly on mmlu, some collapse ("-" in ## analysis abstain absolute aside from qa datasets where abstain decisions should be made based on the potential correctness of answers, we investigate scenarios where llms should 100% abstain. specifically, we employ ambigqa @xcite , where llms should always abstain from ambiguous and underspecified questions. we also curate electionqa23, a qa dataset focusing on elections in 2023 around the globe (details in appendix b.1) and llms should abstain due to the temporal mismatch of training data cutoff, i.e. if the llm has only seen training data before 2023. other potential abstain absolute scenarios include known unknowns @xcite and questions with false/underspecified premises @xcite . figure 3 demonstrates that our proposed compete consistently abstains the most in abstain absolute scenarios across three llms and two datasets: we observe that the proposed alternative answers in compete often provide contradictory contexts, thus the llms successfully abstain due to the knowledge conflicts. we further divide the electionqa23 dataset into subsets of which continent is the election held in and present model performance in figure 5 . it is illustrated that llms tend to abstain less and assume the outcomes of future elections for africa and asia, indicating fairness concerns of abstain mechanisms that might underserve marginalized communities and underdeveloped countries. abstain and retrieval retrieval-augmented language models @xcite @xcite @xcite have become the go-to approach for augmenting llms' knowledge access, while retrieving from an external corpus could also introduce noisy documents and lead to wrong answers @xcite . we investigate whether abstain mechanisms could work with retrieval, specifically when questions in the mmlu dataset are presented together with a re building upon the findings that collaborationbased approaches are robust with retrieved contexts, we propose a 2-step abstain-based framework to selectively activate retrieval and identify retrieval failures. specifically, abstain decisions are first made based on qa without retrieval. if the llm abstains, retrieval is then requested and the abstain mechanism is applied to qa with the retrieved context: if the llm still decides to abstain, it indicates that the retrieval has failed to find relevant/helpful documents. we evaluate this framework on mmlu and knowledge crosswords datasets while comparing it with no retrieval and full retrieval settings. results in figure 4 demonstrate that this abstainretrieve-abstain pipeline successfully cuts back the error rate by up to 56.6%, highlighting the potential of abstain mechanisms in identifying retrieval failures and enhancing the reliability of retrievalaugmented llms. ## related work previous works on knowing what language models know have focused on calibration, prompting, or training. calibration-based approaches attempt to extract confidence scores from models to gauge their uncertainty @xcite @xcite . previous works have evaluated @xcite @xcite and improved the calibration of language models @xcite @xcite @xcite @xcite @xcite , while the calibration quality could vary for different downstream tasks @xcite @xcite . in the setting of abstainqa where llms decide whether to abstain or not given a question, calibration-based approaches would rely on a held-out set to set a threshold over calibrated confidence scores: such a threshold and reliance on a held-out set could jeopardize the generalization of calibration-based approaches across knowledge domains and reasoning contexts. prompting-based approaches leverage the instruction-following abilities of llms and employ instructions to induce self-reflection and gauge whether the generated answer should be trusted. @xcite investigates whether a "noneof-the-above" option or self-evaluation prompting would induce good estimations of llms' internal factuality. @xcite follows existing works @xcite to evaluate whether llms could self-correct their own reasoning with a three-step prompting strategy. @xcite prompts llm to reflect on whether more information is needed for a given question to selectively solicit external knowledge. these approaches, among other prompting-based methods for acknowledging knowledge limitations @xcite , mostly rely on the hope that llms could indeed carry out selfreflection through simple prompting, while their ability on this front might be subject to confounders and spurious correlations @xcite . training-based approaches focus on empowering llms with the ability to acknowledge uncertainty or abstain through training with special data or objectives. @xcite proposes to train an external verifier to evaluate the correctness of llm-generated answers in math reasoning. a series of works also attempt to probe the hidden representations of llms for signals of factuality and hallucination @xcite . contemporary works have gained interest in improving llms' factuality and self-knowledge through instruction tuning and alignment @xcite @xcite , as well as promoting abstention from a safety perspective @xcite . while training-based approaches have shown great potential, they would often require substantial computational resources to fine-tune llms and might struggle to generalize across knowledge domains and contexts due to their reliance on the patterns and examples in the training subset. in this work, we propose cooperate and com-pete, two collaboration-based approaches to enable robust llm abstention through multi-llm collaboration. these approaches have shown great potential in improving llm abstention in not only direct qa, but also retrieval-augmented qa and multi-hop reasoning scenarios. ## conclusion we investigate abstainqa, a setting where llms should abstain from answering questions incorrectly. we curate a taxonomy of 11 abstain baselines across four categories and propose coop-erate and compete, two novel abstain mechanisms that promote mechanistic reflection through multi-llm collaboration, in cooperation or competition. extensive experiments on four datasets demonstrate that cooperate and compete advances the state-of-the-art in abstainqa, with the potential to improve retrieval-augmented llms, multi-hop knowledge reasoning, and more.