Ranked Voting based Self-Consistency of Large Language Models (2025)

Abstract: Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest ”self-consistency” among different reasoning paths (Wang et al., 2023). However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. Code and logs will be released.

## introduction large language models (llms) have shown strong performance in recent years @xcite @xcite @xcite . chain-of-thought prompting @xcite further improves the performance of llms in commonsense @xcite and mathematical @xcite reasoning tasks. building on these advancements, @xcite propose a majority voting based the answer is (b). the answer is (a). the answer is (a). the answer is (c). self-consistency approach, which leverages multiple reasoning paths through sampling to identify the most self-consistent answer, thereby improving the reasoning performance of llms. an example of majority voting based selfconsistency is shown in figure 1(a) . in four responses by the model, option (a) was answered twice, (b) and (c) were each answered once, therefore (a) is considered the answer with the highest "self-consistency". in each response, the model replies with only one option, thereby omitting the possibility and priority of other options, which may introduce biases in the following majority voting process. in this work, we consider obtaining ranked answers instead of only a single answer in each response and employ ranked voting based selfconsistency from multiple ranked answers, which we hope can lead to more reliable self-consistency and better reasoning performance. we show the example of ranked answers in figure 1 @xcite . the model generates the possibility ranking of all options each time. in this case, we find that option (b) ranks in the top two positions in all four responses, while option (a) ranks last in two responses. considering the overall results, option (a) may not be the best suitable final answer. to further decide the best answer according to the ranking information, we attempt to use ranked voting to determine the most appropriate answer based on the obtained ranked answers in responses. ranked voting has been widely used in election systems. specifically, we use three ranking voting methods to facilitate the final answer, including instant-runoff voting @xcite , borda count voting @xcite , and mean reciprocal rank voting. the first two methods are widely used in elections, while the latter one is related with the ranking-based mrr metric. we consider these ranked voting based approaches, compared with majority voting on single answers, can provide more reliable final answer by leveraging the ranking information of ordered answers. we validate our ranked voting based selfconsistency by using advanced llms, including four open-source models llama-3 @xcite , qwen-2.5 @xcite , @xcite , @xcite , and two closed-source models gpt-3.5 @xcite and @xcite . empirical results show that our method consistently outperforms baselines on three multiple-choice and three open-ended question-answering datasets. ## method we first introduce the majority voting based selfconsistency baseline ( §2.1) and propose obtaining the ranked answers in each llms' response ( §2.2). then we provide detailed descriptions to three ranked voting methods as shown in figure 2 , including instant-runoff voting ( §2.3), borda count voting ( §2.4), and mean reciprocal rank voting ( §2.5). in the final we briefly present overall comparison and how to handle tie votes ( §2.7). ## analyses we further analyze our proposed method from different perspectives. unless otherwise specified, we conduct experiments on commonsenseqa and wikidata using llama-3-8b. the impact of k. figure 3 shows the performance of different number of samples k. for common-senseqa, under different settings of k, ranked voting methods consistently outperform majority voting based self-consistency. moreover, compared with baseline with @xmath0, our method achieves better results when @xmath1, and its performance improves as k increases. in contrast, the baseline shows limited improvement as k increases. for wikidata, the results of instant-runoff voting and baseline are similar, while the other two methods consistently achieve an accuracy improvement of 0.5∼1.0 with different settings of k. ## conclusion we introduce ranked voting based self-consistency to enhance the reasoning performance of large language models. our method outperforms traditional majority voting techniques by generating and incorporating ranking information among multiple candidate answers during the reasoning process. the proposed method improves the reliability of self-consistency and boosts the performance. extensive experiments demonstrate the effectiveness of our method, as it consistently outperforms baselines across a variety of multiple-choice and open-ended question-answering datasets, using both open-source and closed-source llms.