Ask Again, Then Fail: Large Language Models’ Vacillations in Judgment (2024)

Abstract: We observe that current large language models often waver in their judgments when faced with follow-up questions, even if the original judgment was correct. This wavering presents a significant challenge for generating reliable responses and building user trust. To comprehensively assess this issue, we introduce a Follow-up Questioning Mechanism along with two metrics to quantify this inconsistency, confirming its widespread presence in current large language models. Furthermore, to mitigate this issue, we explore various prompting strategies for closed-source models, and develop a training-based framework Unwavering-FQ that teaches large language models to maintain their originally correct judgments through synthesized high-quality preference data. Our experimental results confirm the effectiveness of our framework and its ability to enhance the general capabilities of large language models.

## introduction generative large language models (llms) like @xcite are considered the latest breakthrough technology, having progressively integrated into people's daily lives and found applications across various fields @xcite @xcite . despite their remarkable capabilities in generating relevant responses to user inquiries, we find that they often start to falter in their judgments when users continue the conversation and express skepticism or disagreement with the model's judgment. this leads to responses that significantly deviate from previous ones, even if the model's original judgment is accurate. this work refers to it as llms' judgment consistency, which pertains to the model's vacillation in judgments on objective questions with fixed answers. however, we emphasize that the current level of attention to this issue is still insufficient, even though a few recent studies have identified this issue from specific perspectives @xcite . in this work, we argue that there are still two main challenges regarding this issue: (1) how to comprehensively assess the judgment consistency issue and employ appropriate metrics to quantify it accurately; (2) how to mitigate this issue through technical means, whether for open-source or proprietary models. our research endeavors are centered on addressing these two pivotal challenges. for the first challenge, inspired by the theory of "questioning strategies" in education @xcite as demonstrated in figure 1 , we design a follow-up questioning mechanism with two metrics to investigate the judgment consistency of conversational llms systematically. this mechanism is conceptually derived from the teaching process, where teachers extend the dialogue through additional queries, negations, or misleading prompts following a student's response, aiming to ascertain the depth of their understanding. ## evaluation of llms' judgment consistency to accurately evaluate and quantify llms' judgment consistency, we design the follow-up questioning mechanism with two metrics. after the model initially answers correctly, we continue dialogues to question, negate, or mislead it, then observe any judgment changes. ## llms waver in judgements as main results, we analyze chatgpt's judgement consistency in arithmetic ( § 4.1), commonsense ( § 4.2), symbolic ( § 4.3), and knowledge reasoning tasks ( § 4.4), respectively. subsequently, we extend our validation of this issue to other llms under the same settings ( § 4.5). ## towards mitigating the inconsistency essentially, we believe this issue may stem from biases in the data collection and annotation process, such as human annotators possibly favoring seemingly correct but sycophantic answers. @xcite . ideally, a conversational assistant should maintain confidence in its judgments and not change its stance when questioned, while also being able to recognize and correct errors upon further questioning. achieving a balance between these two aspects is challenging, with limited research currently addressing this. in this work, we explore various strategies to mitigate this issue, including training-free and training-based ones. for closed-source models, we explore training-free methods, namely by adjusting prompts to alleviate the issue. for open-source models, we introduce a training-based framework named unwavering-fq to help the model maintain its initial correct judgments unwaveringly and correct errors. ## related work for a broader range of related work, refer to appendix c due to limited space. alignment aims to teach language models to follow instructions, align with human values and intention @xcite and avoid hallucinations @xcite . the judgment consistency issue we reveal represents unaligned aspects within current language models. relatedly, @xcite initially explored this issue through debates between models. distinguishing our work, we conduct a comprehensive evaluation by introducing the follow-up questioning mechanism to make the issue more transparent, and then introduce holistic solutions to significantly alleviate it. sycophancy manifests as models excessively aligning with and indulging incorrect human viewpoints. preliminary research has explored this issue @xcite . @xcite introduce a simple method of data synthesis using fixed templates to mitigate sycophancy, especially targeting multiple-choice questions. the issue revealed in this work is closely related to sycophancy, yet we also uncover a new phenomenon: models exhibit caution and neutrality in the face of disturbances, a behavior not extensively studied, as described in error analysis (cf. § 2.4.3). moreover, our framework synthesizes preference data with language models for multi-turn dialogues, not confined to any specific task. calibration and honesty involve how models express uncertainty in their responses @xcite and the consistency of their replies with their inherent knowledge @xcite . our follow-up questioning is predicated on the correct initial response of the model, implying the model possesses relevant intrinsic knowledge and reasoning capabilities. if the model's judgment significantly wavers in response to follow-up questions, it indicates insufficient alignment in this aspect. our work is dedicated to thoroughly assessing and mitigating this issue. prompt robustness refers to how different prompts affect model responses @xcite @xcite . we find language models lack robustness to follow-up prompts. relatedly, some studies have shown that incorporating additional context into prompts significantly impacts performance @xcite . unlike these evaluative studies, our focus is on conversational scenarios, for which we have developed effective mitigation strategies. beyond prompting-based approaches, we also propose a training-based framework for this issue. ## conclusion the wavering of large language models' judgments when confronted with follow-up questions poses a considerable obstacle to generating reliable responses and establishing user trust. this work focuses on how to comprehensively assess judgment consistency and mitigate this inconsistency issue. inspired by questioning strategies in education, we propose the follow-up questioning mecha-nism and two metrics to systematically access the judgment consistency across models (including proprietary and open-source models). we explore both training-free prompting methods and a trainingbased framework unwavering-fq to mitigate this issue, with experimental results showing significant improvement. we aspire for our work to be beneficial to future research in this direction.