Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024)

Abstract: Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. Prior works utilize verification properties in software engineering to verify and re-rank solutions in a majority voting manner. But the assumption behind them that generated verification properties have better qualities than solutions may not always hold. In this paper, we treat them equally as different perspectives of LLMs’ reasoning processes. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency across outputs from multiple perspectives. Specifically, we prompt LLMs to generate diverse outputs from three perspectives, Solution, Specification and Test case, constructing a 3-partite graph. With two measure functions of consistency, we embed both inter- and intra-consistency information into the graph. The optimal choice of solutions is then determined based on analysis in the graph.MPSC significantly boosts performance of foundation models (ChatGPT in this paper) on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%), even surpassing GPT-4.

## introduction in recent years, pre-trained large language models (llms) have demonstrated unprecedented proficiency in understanding, generating, and reasoning with human language @xcite @xcite . among the diverse applications of llms, code generation stands out as pivotal task and has been acknowledged as a fundamental task for benchmarking @xcite . this task entails models to generate source codes from provided natural language intents. many foundation models have exhibited remarkable zero-shot performance in code generation, such as chatgpt and gpt4 (openai, 2023), with successful deployments in real-world applications like github copilot. despite the remarkable abilities, llms often struggle to generate the correct code in a single attempt. therefore, previous works sample diverse codes from llms and re-rank them by introducing verification properties from software engineering. for example, @xcite generates test cases as a verification property, while algo @xcite generates oracles, the brute-force version of desired algorithms, as a verification property. these methods are essentially variants of majority voting, making the assumption that the correctness of experts (i.e. the verification properties) is better than that of choices (i.e. the desired code outputs). however, both verification properties and desired code outputs are usually generated by the identical model with respect to the same question, and hence the preference over verification properties is not always correct. instead, we believe that both desired code outputs and verification properties should be treated equally, since they are different perspectives of llm's deliberate thinking process in face of identical questions. aggregating various outputs from different perspectives can lead to a more credible result. to achieve this, we propose the multi-perspective self-consistency (mpsc) framework that incorporates both inter-consistency across multiple perspectives and intra-consistency within a single perspective. in this way, mpsc can fully leverage the consistency information within llms and select the model output with the most consistent functionality as the final answer. in our framework, various verification techniques from software engineering can be included as extended perspectives for better reasoning. specifically, we prompt the llm to simultaneously generate diverse outputs from three wellestablished perspectives in software engineering, namely solution, specification and test case @xcite . solutions implement the desired functionality, specifications demonstrate the intended properties in formal language, while test cases outline the expected behavior for some specific inputs. then, we treat these model outputs as vertices in a graph, and establish connections (i.e. edges) based on the pairwise agreement of vertices from different perspectives. our goal is to identify the most reliable output using a score function, which evaluates all vertices by considering both intra-and inter-consistency information encoded in the graph. specifically, the intra-consistency information guides the function to favor the most internally consistent output within a single perspective, while inter-consistency ensures that the scores for two outputs from different perspectives are similar if they reach a consensus. we formalize the learning process of the score function as an optimization problem adhering to these two consistency criteria and leverage an iterative algorithm proposed by @xcite to achieve this goal. we evaluate mpsc on four widely used code generation benchmarks, including humaneval @xcite , humaneval+ @xcite , mbpp @xcite and @xcite . experimental results show that our method boosts the performance of chatgpt by a large margin, 15.91% in humaneval, 15.64% in humaneval+, 6.43% in mbpp and 9.37% in code-contests. our framework even surpasses gpt-4 (openai, 2023) as shown in figure 1 . ## multi-perspective self-consistency a single perspective can often lead to an incomplete understanding of a problem, akin to the parable of "blind men describing an elephant". the reasoning process of llms follows a similar pattern. llms generally cannot guarantee the correctness of generated output in a single attempt, especially in code generation, which necessitates proficient natural language understanding, deliberate reasoning and rigorous controllability. however, a key aspect of human intelligence is the ability to think from multiple perspectives, resulting in a more comprehensive understanding of situations and more accurate solutions to problems. inspired by the human cognition process, we propose a novel code generation method by reasoning from three well-established perspectives, solutions, specifications and test cases. although noisy outputs may inevitably be included in the generated outputs of every perspective, we can leverage both intra-consistency and inter-consistency among the diverse outputs to distinguish the best ones from the noise. an overview of our proposed mpsc method is illustrated in figure 2 . ## related work prompting techniques on consistency. based on chain-of-thought mechanism @xcite , previous works have adopted various prompting techniques and decoding strategies to reveal the consistency of llm outputs and further enhance the capabilities of llms. one line of approaches decodes multiple times from the same perspective and aggregate the results @xcite @xcite @xcite . for example, wang et al. (2022) targets tasks with fixed answer sets and scores each answer based on the output frequency. building on this, sun et al. (2022) introduces recitation as context for augmentation. while jung et al. (2022) focus on the two-value entailment relations (true or false) between statements and explanations. they treat the inference process as a weighted max-sat problem and utilize a logistic solver to solve it. another line draws inspiration from the "dual process" in cognitive science, which posits that human reasoning is dominated by system 1 and system 2 (daniel, 2017; sloman, 1996). as a result, these approaches require llms to play different roles like generator (i.e. system 1) and verifier (i.e. system 2), and optimize the result iteratively by a conversational way (madaan et al., 2023; shinn et al., 2023; zhu et al., 2023). xiong et al. (2023a) also proposes the concept of "inter-consistency". instead of referring to the consistency within the same llm, they focus to tackle the inter-inconsistency problem between different models with a formal debate framework. llm for code. llms pretrained on large-scale code data have demonstrated strong capabilities in the field of code generation, such as codex (chen et al., 2021), alphacode (li et al., 2022), code-gen (nijkamp et al., 2023), incoder (fried et al., 2022), starcoder (li et al., 2023), code llama (rozière et al., 2023), wizardcoder (luo et al., 2023), deepseekcoder (guo @xcite . however, they remain unreliable, particularly in scenarios involving complex input-output mappings. because of the low tolerance of compilers and operating systems for bugs, the instability makes llms hard to deploy into real-world applications. several methods have been proposed to mitigate the phenomenon @xcite @xcite @xcite @xcite @xcite . the line of work with the most direct relevance to ours is to re-rank generated solutions in a post-hoc manner. for example, @xcite matches the execution results of generated solutions for minimum bayes risk selection. @xcite prompts another model as a reviewer to check whether generated programs satisfy the given language instruction by measuring p(instruction|program). @xcite additionally generates test cases to verify the generated solutions. similarly, algo @xcite additionally generates exhaustive search algorithms as oracle programs to generate high quality test cases for verification. ## conclusion in this paper, we present a novel code generation method, multi-perspective self-consistency (mpsc), aimed at enhancing the performance of llms in complex code generation tasks where a single attempt may not suffice to ensure the accuracy of the output. our proposed mpsc strategy capitalizes on both intra-and inter-consistency across three perspectives, solutions, specifications and test cases, to identify the most reliable answer. we systematically validate the effectiveness of mpsc through comprehensive experiments conducted on four widely used code generation benchmarks. the evaluation results demonstrate that mpsc outperforms existing methods and achieves the state-of-the-art performance on all of them.