Self-Training Meets Consistency: Improving LLM s’ Reasoning with Consistency-Driven Rationale Evaluation (2025)

Abstract: Self-training approach for large language models (LLMs) improves reasoning abilities by training the models on their self-generated rationales. Previous approaches have labeled rationales that produce correct answers for a given question as appropriate for training. However, a single measure risks misjudging rationale quality, leading the models to learn flawed reasoning patterns. To address this issue, we propose CREST (Consistency-driven Rationale Evaluation for Self-Training), a self-training framework that further evaluates each rationale through follow-up questions and leverages this evaluation to guide its training. Specifically, we introduce two methods: (1) filtering out rationales that frequently result in incorrect answers on follow-up questions and (2) preference learning based on mixed preferences from rationale evaluation results of both original and follow-up questions. Experiments on three question-answering datasets using open LLMs show that CREST not only improves the logical robustness and correctness of rationales but also improves reasoning abilities compared to previous self-training approaches.

## introduction large language models (llms) can enhance multistep reasoning abilities by generating intermediate reasoning steps (i.e., rationale) before arriving at an answer @xcite . training llms on high-quality rationales has been shown to improve their reasoning capabilities @xcite @xcite . therefore, collecting high-quality rationales is becoming increasingly important for training the reasoning abilities of llms. however, due to the high cost associated with collecting high-quality rationales, self-training approaches have emerged, focusing on (q) which type of activity would most likely be included on a weather map? (𝑟 1 ) weather maps show weather phenomena. hurricanes are weather phenomena, and the others are not. therefore, the answer is d. therefore, the answer is d. llm a. satellite b. seismic c. volcanic d. hurricane 𝑟 1 is a the answer to q? no 𝑟 1 is b the answer to q? no 𝑟 1 is c the answer to q? no 𝑟 1 is d the answer to q? yes 𝑟 2 yes no 𝑟 2 𝑟 2 𝑟 2 yes yes let llm solve follow-up questions with each generated rationale (𝑟 1 , 𝑟 2 ) is a the answer to q? is b the answer to q? is c the answer to q? is d the answer to q? in self-training approaches, accurately evaluating the quality of generated rationales is essential. previous studies have evaluated rationale quality by examining whether the generated rationales lead to the correct answer to a given question @xcite @xcite @xcite . however, using the correctness of a single prediction is unstable, as llms can reach correct answers through inappropriate reasoning steps @xcite . figure 1 shows an example of two generated rationales, r 1 and r 2 . despite r 2 shows incomplete reasoning, previous approaches would consider both rationales equally appropriate since they both lead to the correct answer for q. training models on such inappropriate rationales can cause them to learn flawed reasoning patterns. to address this problem, we propose crest (consistency-driven rationale evaluation for self-training), a novel framework for llm self-training. the core idea of crest is to further evaluate rationales using follow-up questions that ask whether each answer option in the original question is correct or not. we first generate diverse rationales using temperature sampling and evaluate them with an llm as shown in figure 1 . subsequently, we train the llm on these rationales, rewarding rationales that lead to more consistent predictions (i.e., r 1 ) and penalizing those that lead to less consistent predictions (i.e., r 2 ). to achieve this, we propose two methods: rationale filtering and preference learning. in rationale filtering, we remove rationales that lead to incorrect answers in more than a certain number of follow-up questions during the supervised fine-tuning process. in preference learning, we train the model on mixed preferences from results of both original and follow-up questions, to favor rationales that result in correct answers in a greater number of follow-up questions. we conduct experiments on three natural language reasoning question-answering datasets, including reclor @xcite , arc @xcite , and csqa @xcite . we compare crest to other self-training approaches using llama 3 model (ai@meta, 2024) and gemma model @xcite . our findings show that crest can train an llm to generate more correct and robust rationales, improving its reasoning performance. our contributions are as follows: ## consistency-driven rationale evaluation for self-training this section describes our approach, consistencydriven rationale evaluation for self-training (crest) which trains reasoning abilities through consistency-driven rationale evaluation with follow-up questions. ## experiments this section describes the experiments and results of crest compared to other self-training approaches. first, we introduce the three datasets used for model training and testing. next, we present the experimental setup, including the base llm, key hyperparameters, and performance metrics. we also introduce the baseline approaches used for comparison, and finally, we present the results of the experiments. ## analysis in this section, we explore the effectiveness of consistency-driven evaluation and the impacts of rationale filtering and preference learning in crest on model performance, through analyses using the llama 3 8b model as the base model. our analysis includes examining the correlation between z and z and conducting ablation studies on parameters such as t and λ to assess how the proposed methods in crest contribute to performance improvement. to investigate the impact of preference learning with p z , we create a model that trains m sft using preference learning with only p z , which we refer to as m sft /w pz , and compare it to m crest . ## conclusion in this paper, we propose crest, a novel selftraining framework that evaluates generated rationales in a fine-grained manner by letting the llm solve follow-up questions derived from the original question. we propose two methods for utilizing the evaluation results in training: filtering out less consistent rationales for supervised fine-tuning and employing preference learning to favor more consistent rationales over less consistent ones. experimental results on three question-answering datasets show that crest enables an llm to generate more correct and robust rationales and achieves better performance compared to previous approaches.