Rationale-Aware Answer Verification by Pairwise Self-Evaluation (2024)

Abstract: Answer verification identifies correct solutions among candidates generated by large language models (LLMs). Current approaches typically train verifier models by labeling solutions as correct or incorrect based solely on whether the final answer matches the gold answer. However, this approach neglects any flawed rationale in the solution yielding the correct answer, undermining the verifier’s ability to distinguish between sound and flawed rationales. We empirically show that in StrategyQA, only 19% of LLM-generated solutions with correct answers have valid rationales, thus leading to an unreliable verifier. Furthermore, we demonstrate that training a verifier on valid rationales significantly improves its ability to distinguish valid and flawed rationale. To make a better verifier without extra human supervision, we introduce REPS (Rationale Enhancement through Pairwise Selection), a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions. Verifiers trained on solutions selected by REPS outperform those trained using conventional training methods on three reasoning benchmarks (ARC-Challenge, DROP, and StrategyQA). Our results suggest that training reliable verifiers requires ensuring the validity of rationales in addition to the correctness of the final answers, which would be critical for models assisting humans in solving complex reasoning tasks.

## introduction reasoning in large language models (llms) @xcite @xcite plays a vital role in their intelligent behavior @xcite . recent advancements, owing to progressive scaling @xcite and high-quality corpora @xcite , facilitate llms in solving complex reasoning tasks including mathematical reasoning and multi-hop reasoning. further improvements are possible by prompting llms to generate intermediate reasoning steps that validate why the generated answer is correct @xcite . however, llm-generated reasoning often exhibits factual and logical consistency error @xcite @xcite @xcite . erroneous reasoning can be mitigated using a trained verifier model @xcite to select the best solution from multiple generated solutions, each consisting of an answer and its intermediate reasoning (rationale). this methodology delivers more accurate answers than a single-answer setting @xcite @xcite . however, existing studies mainly define the training labels for the verifier by solely checking the final answer. this approach may lead to a verifier that prefers faulty reasoning when llms generate the correct answers for wrong reasons (figure 1 ). this is particularly problematic when models assist humans in complex tasks @xcite , wherein humans can not fully examine the model's feedback. this raises two questions: (1) how often do models generate correct answers with flawed rationales? (2) does training on flawed rationale as positive samples yield an untrustworthy verifier that cannot discriminate between flawed and valid rationale? we investigate these questions using strategyqa @xcite as the testbed (detailed in section 3). we use @xcite to assess the validity of the rationales based on their factual and logical consistency with the annotated factual information. for the first question, we find that although 59% of the model-generated solutions contain the correct answer, only 19% of these correct-answer solutions are judged as having valid rationale, which is also supported by our manual inspection. for the second question, we create training datasets with varying levels of reasoning quality for the verifier. the verifier's accuracy in selecting valid reasoning significantly increases as the quality improves. to enhance the verifier's ability to discern the validity of rationales, we introduce reps (rationale enhancement through pairwise selection), a training method that leverages llms' pairwise selfevaluation to select high-quality rationales from candidates. by refining the training data with these valid rationales, reps enables to train more reliable and rationale-aware verifiers (section 4). experiments across three reasoning datasets, arc-challenge @xcite , drop @xcite , and strategyqa demonstrate that verifiers trained on reps consistently prefer solutions with valid rationales compared to those trained on datasets labeled solely based on the final answer's correctness. while reps serves as an effective quality filter for rationales without extra human supervision, our analysis implies that excessive pairwise evaluation may amplify inherent preference biases in llms, such as favoring longer text (sections 5 and 6). we hope future work will explore methods to mitigate these biases and further enhance the reliability of verifiers, in preparation for models becoming capable of solving tasks that humans cannot examine. this work takes the first step towards building trustworthy verifiers that can discern the validity of rationales. our contributions can be summarized as follows: our analysis reveals that a large portion of correct answer solutions have invalid reasoning, and verifiers trained on such rationales tend to select solutions with similarly faulty reasoning. ## does flawed rationale lead to untrustworthy verification? llms may generate responses that contain correct answers but invalid reasoning paths @xcite . thus, labeling generated solutions as positive samples based on the correctness of the final answer, without considering the validity of the reasoning, may yield a verifier that fails to distinguish between sound and flawed reasoning. herein, we investigate this hypothesis by decomposing it into two questions: (1) how often do llms generate correct answers with invalid reasoning? (section 3.3) (2) how does the rationale quality in the positive training data affect the verifier's performance in selecting valid reasoning? (section 3.4) ## rationale enhancement through pairwise selection (reps) as shown in section 3, a large portion of llmgenerated answers contain flawed reasoning, even if the final answer is correct. moreover, increasing the ratio of valid rationales in the positive samples improves the verifier's ability to identify sound reasoning. thus, filtering out solutions with flawed rationales is crucial for building a reliable verifier. we propose reps, which uses llms' pairwise comparison to iteratively select high-quality rationales from diverse candidates. the overall process of reps is illustrated in figure 4 . for a question q, we first prompt the generator model m g to produce a set of candidate solutions @xmath0, s 2 , . . . ,, where each solution s i consists of an answer a i and a rationale r i . from this set, we select n solutions with correct answers a + to form the candidate solution set for the tournament-style pairwise evaluation. we use tournament-style pairwise evaluation to find the best rationale among the n candidates. in each round, the generator model m g acts as the evaluator that compares the rationales (r i , r j ) of two solutions (s i , s j ) s times and selects the more factually grounded and logically consistent one as the winner through majority voting. the evaluator is provided with the question, answer, two candidate rationales, and (depending on the dataset) passage and answer options. given hand-crafted few-shot exemplars, the evaluator model outputs a justification for which rationale is better and why, followed by the preferred rationale's index (1 or 2). 4 to mitigate the position bias reported in llmbased evaluations @xcite , we alternate the presentation order of rationale 1 and 2 across the s comparisons within each round. we perform this pairwise comparison for all n 2 pairs of solutions. after the pairwise comparisons, the n 2 preferred rationales form a new set of candidate solutions s ′ , and the pairwise comparialgorithm 1 rationale enhancement through pairwise selection (reps) ## discussion the discrepancy between rationale accuracy and task performance while reps effectively improves rationale accuracy of verifiers, it does not significantly enhance overall task performance. this aligns with section 3.3 and recent findings suggesting a weak link between rationale quality and answer correctness @xcite . this unfaithfulness becomes particularly critical in the context of scalable oversight @xcite , where humans may find it challenging to evaluate the correctness of the model's outputs. in such scenarios, humans might be misled by seemingly plausible yet unfaithful rationales. future research should focus on improving rationales regarding quality and faithfulness. ## conclusion we investigate how rationale quality affects a verifier's ability to select valid answers. we empirically demonstrate that many model-generated rationales contain errors, even when the final answer is correct, leading to untrustworthy answer verification. we introduce reps, a method that uses llms' pairwise comparison to iteratively refine generated rationales. experiments on three reasoning datasets show that reps significantly outperforms baseline verifier models, particularly in selecting solutions with valid rationales. our analysis shows iterative pairwise evaluation improves rationale quality but may amplify llm-based evaluators' biases.