Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection (2024)

Abstract: Self-detection for Large Language Models (LLMs) seeks to evaluate the trustworthiness of the LLM’s output by leveraging its own capabilities, thereby alleviating the issue of output hallucination. However, existing self-detection approaches only retrospectively evaluate answers generated by LLM, typically leading to the over-trust in incorrectly generated answers. To tackle this limitation, we propose a novel self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers. It thoroughly compares the trustworthiness of multiple candidate answers to mitigate the over-trust in LLM-generated incorrect answers. Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each candidate answer, and then aggregates the justifications for comprehensive target answer evaluation. This framework can be seamlessly integrated with existing approaches for superior self-detection. Extensive experiments on six datasets spanning three tasks demonstrate the effectiveness of the proposed framework.

## introduction large language model (llm) typically suffers from the hallucination issue, @xcite @xcite , which significantly undermines the trustworthiness of llm's outputs. a promising research direction for evaluating the output trustworthiness and identifying incorrect outputs is self-detection @xcite @xcite . given a question, self-detection aims to leverage llm's own ability to evaluate the trustworthiness of its generated answers, without relying on external knowledge sources or specifically trained detection models. this paper investigates self-detection methods tailored for black-box api because it provides a clear and direct explanation of how to repair a torn shirt... ## think twice before assure framework implementing the proposed paradigm involves two key considerations. first, given the potential bias of llm over-trust in the generated answer a, it is essential to develop strategies to resist this bias and thoroughly evaluate the trustworthiness of each answer a q i . secondly, it is crucial to derive strategies to effectively combine these evaluations for effective self-detection of a. to address these concerns, we introduce the following two-step framework. step 1: reflection and justification. we first instruct llm to reflect on the trustworthiness of each answer a q i and force llm to seek justification for a q i as the correct answer of q, as defined by eq. 8. the llm is instructed with the prompt p e in ## related work confidence calibration of llm. confidence calibration has been previously studies in neural networks @xcite and applied in nlp models @xcite @xcite . after the advent of llm, many confidence calibration methods utilize the output token probability, such as semantic uncertainty @xcite , temperature scaling @xcite , entropy-based @xcite , semantic significance @xcite , and fine-tuning for calibration @xcite . @xcite also employ model ensemble for calibration. our research is orthogonal to them, since we focus on black-box api llm itself. other recent work suitable for black-box llm includes fidelity elicitation @xcite , fact elicitation @xcite and perturbation generation @xcite . self-evaluation of llm. llm self-evaluation often focuses on specific domains, e.g., code generation @xcite , natural language generation @xcite and fact checking @xcite . the general methods include p(true) @xcite and directly asking llm @xcite . @xcite 2024 ) also performs answer reflection and employs model collaboration, yet they still focus on answers generated by llm. note that self-detect @xcite is also a general self-evaluation method following the three-step confidence calibration pipeline. other works that are related but orthogonal to us include training independent models for llm evaluation @xcite @xcite @xcite , and using external tools for llm verification @xcite . they are usually applied to specific domains, while we aim at llm self-detection for general tasks. also, finetuning llm for better trustworthiness @xcite is orthogonal to us. application of llm self-detection. the outcome of self-detection can be applied in many ways to avoid hallucination and erroneous outputs, such as identifying potentially hallucinated generation for knowledge retrieval and verification @xcite , guided output decoding @xcite , identifying ambiguous questions @xcite , selective generation @xcite , and llm self-improve @xcite . more applications can be found in this survey @xcite . ## conclusion in this paper, we tackled the over-trust issue of self-detection on black-box api llms. we categorized existing methods into two paradigms and pointed out their limitation of merely evaluating on llm-generated answer with potential llm overtrust. we proposed a novel paradigm to address this limitation by comprehensively evaluating the trustworthiness of multiple candidate answers in the answer space. following our paradigm, we presented a two-step framework t 3 by asking llm to reflect and justify the validity of each answer for joint confidence calibration. our framework achieved improved self-detection performance over compared methods and was combined with existing methods for further improvement. in future work, we will explore the combination of t 3 with more methods, and its utility in white-box llms.