Paper: Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (ICLR-2024)

Reviewer comment (Novelty):
Novelty: [1] already proposed to construct a graph and explore more fine-grained consistency via MAX-SAT solving (although for natural language based reasoning tasks). The similarities and differences should be better discussed in the current paper.  
[1] Jung et al. Maieutic prompting: Logically consistent reasoning with recursive explanations. EMNLP 2022.

Author response:
> [1] already proposed to construct a graph and explore more fine-grained consistency via MAX-SAT solving

Thanks for your suggestions about the missing related work.

Acknowledging that both MPSC and Maieutic Prompting are post-hoc methods aimed at enhancing the reasoning abilities of LLM, it is crucial to highlight the distinctions between them.

The primary difference lies in motivation. Maieutic Prompting addresses the inconsistency between the explanation and label generated by LLM, focusing on **the two-value entailment relation (T/F) between a statement and an explanation**. In contrast, MPSC is designed to fully exploit **the consistency among  outputs from different perspectives** within LLM. In our opinion, MPSC covers a broader scope, as it is straightforward to regard the statement and explanation as two perspectives and leverage the entailment relation as a form of inter-consistency measure within our framework. 

Furthermore, there are many differences in the specific method design. Maieutic Prompting regards the generated outputs as a tree structure, emphasizing the entailment between each parent and son. Therefore, its outputs has a sequential causal relationship. In contrast, in our approach, outputs from different perspectives has no order. And we construct a graph to encode pair-wise relations between vertices from different perspectives. Regarding the inference process,  while Maieutic Prompting treats it as a MAX-SAT problem, which is NP-hard, we propose a continuous optimization problem over the graph, benefiting from a closed-form solution. 

Thanks for pointing out this paper. We will add it into the related work session in the updated paper.

Reviewer comment (Experiments):
Designing the perspectives: for each task, how much manual effort is needed to design those perspectives? (and writing the prompts for each perspective?) How sensitive is the final performance w.r.t. the prompts written for each perspective?

Author response:
> how much manual effort is needed to design those perspectives? (and writing the prompts for each perspective?) How sensitive is the final performance w.r.t. the prompts written for each perspective?

In response to the first question about manual designed perspectives, we have made a general response. Please refer to it for more details.

Regarding the other two questions about prompt design, we barely work on prompt design and only use convention natural language instructions (please refer to Table 12/13/14). It is evident that the prompt we used is definitely not (even far from) the optimal. 
Moreover, we emphasize that the prompt design will not only affect MPSC but also other baselines, since they utilize the same set of generated solutions and test cases. 

In our opinion, the two points can guarantee the reproducibility of the reported performance in our paper and the prompt insensitivity of MPSC.

Reviewer comment (Experiments):
Constructing the graph: each edge needs to be specifically designed (section 3.2), why those choices and how much difference does it make if designed differently? For intra-consistency, similarly the authors designed many different measures, and based on the experiment results the best measure varies depending on the task. How would one pick which measure to use in practice?

Author response:
> constructing the graph: each edge needs to be specifically designed (section 3.2), why those choices and how much difference does it make if designed differently? 

The Solution, Specification and Test case are three well-established perspectives for code generation [1]. Based on their definitions, the inter-consistency measure through code execution is pretty natural and intuitive. We also discuss the possibility of the automatic inter-consistency measure in the general response. Please refer to it for more details.

[1] Agile Software Development Methods: Review and Analysis

> For intra-consistency, similarly the authors designed many different measures, and based on the experiment results the best measure varies depending on the task. How would one pick which measure to use in practice?

We designed many different measures to better analyze different types of intra-consistency within LLM (e.g. *Bayes Risk* is related to lexical consistency, *Cardinality* is designed to evaluate semantic consistency). Across various scenarios, we consistently observed robust performance with MPSC-Weighted Cardinality, making it our recommended preference.

Reviewer comment (Theory):
Solving the optimization: depending on the number of nodes and edges, the solving part could be very expensive; even with the iterative algorithm, it might take many rounds to reach convergence. This point needs to be better discussed.

Author response:
> solving the optimization: depending on the number of nodes and edges, the solving part could be very expensive

The iterative algorithm converges pretty fast. On HumanEval, the average convergence is achieved in 45.21 iterations, while on mbpp, it takes 32.52 iterations. Notably, the algorithm converges even more swiftly on CodeContests, requiring only 11.75 iterations. This consistent pattern of fast convergence across various benchmarks underscores the algorithm's efficiency and effectiveness.

Reviewer comment (Experiments):
Choice of parameters: what is the value of \alpha in experiments? From Figure 4, on human-eval the performance varies a lot depending on \alpha. The final reported performance seems to be the \alpha that achieves the best performance. But without a training/dev set, how could one pick \alpha in an unsupervised way?

Author response:
> without a training/dev set, how could one pick \alpha in an unsupervised way?

First of all, we want to emphasize that even with randomly selected alpha (e.g. 0.05 ~ 0.95), the performance of MPSC (especially the Weighted Cardinality setting) still outperforms baselines (please refer to Figure 4).

We admit that we cannot select the optimal alpha without priori about the test data distribution. In our main experiment, we sample 10 cases (which is a relatively small number) as our dev dataset. We then conduct a grid search for alpha on it (from 0.05 to 0.99 with a step of 0.05).

Reviewer comment (Experiments):
If given the same number of samples, i.e., baselines with 800 samples vs MPSC, how does the performance compare?

Author response:
> if given the same number of samples, i.e., baselines with 800 samples vs MPSC, how does the performance compare?

We totally agree with you about this fair experimental settings. In response to your concerns, we reduce the number of samples we use to 80 solutions, 20 specifications and 100 test cases, which is even unfair for MPSC since we can actually extract 10 test cases per API call. Because of your attention to variance in our method, we conducted sampling with five distinct seeds and reported the mean Pass@1 performance along with the std. The results are shown as follows. It is evident that MPSC still maintains high performance.
- 83.52(±0.72) on HumanEval
- 73.43(±0.43) on HumanEval+
- 8.89(±0.7) on CodeContests
- 72.2(±0.76) on MBPP

Reviewer comment (Evaluation):
What is the variance of the proposed method, given that more samples from different perspectives are drawn?

Author response:
> what is the variance of the proposed method, given that more samples from different perspectives are drawn?

We think the variance is likely to decrease, since the only source of randomness in MPSC comes from the sampling process of LLM. As the number of samples increases, it is reasonable to anticipate a reduction in variation.

Reviewer comment (Novelty):
How does the proposed method compare with [2]? [2] also uses test cases to improve language models' code generation.  
[2] Chen et al. Teaching Large Language Models to Self-Debug. 2023.

Author response:
> how does the proposed method compare with “Teaching Large Language Models to Self-Debug.”

This paper doesn’t release their codes, which makes the reproduction much harder. We have already tried to reproduce the method according to the prompts given in their appendix, but the gains is pretty small on our datasets.

Reviewer comment (Experiments):
Table 4 shows the test cases give the most gains, so maybe a simple baseline could be added: use the generated test cases to filter out incorrect generated solutions, and then apply self-consistency on the filtered set.

Author response:
> Table 4 shows the test cases give the most gains, so maybe a simple baseline could be added: use the generated test cases to filter out incorrect generated solutions, and then apply self-consistency on the filtered set.

Even though the test case perspective gives the most gains in ablation study, it doesn’t guarantee the quality of generated test cases. Notably, the accuracy of generated test cases is relatively low, with only 63.83% on HumanEval and a mere 24.54% on MBPP. Directly filtering solutions through these generated test cases is anticipated to result in subpar performance, potentially even worse than employing vanilla self-consistency without the filtering.

We want to reiterate that the substantial performance gains of MPSC are not attributed to the quality of the LLM generated outputs. Instead, these gains stem from the consistency within the LLM, which is fully exploited by MPSC.

Reviewer comment (Experiments):
How is the value of \alpha chosen for each task? Is it chosen based on the final performance (which means the test data is known when you pick \alpha)?

Author response:
> without a training/dev set, how could one pick \alpha in an unsupervised way?

First of all, we want to emphasize that even with randomly selected alpha (e.g. 0.05 ~ 0.95), the performance of MPSC (especially the Weighted Cardinality setting) still outperforms baselines (please refer to Figure 4).

We admit that we cannot select the optimal alpha without priori about the test data distribution. In our main experiment, we sample 10 cases (which is a relatively small number) as our dev dataset. We then conduct a grid search for alpha on it (from 0.05 to 0.99 with a step of 0.05).

Reviewer comment (Experiments):
How does the proposed method work on the best model experimented (GPT-4)? Does it still give gains?

Author response:
> How does the proposed method work on the best model experimented (GPT-4)? Does it still give gains?

We are sorry that GPT4 API is too expensive to conduct the experiments as you wish. Here we conduct an alternative experiment to improve GPT4 generated solutions with ChatGPT generated test cases and specifications. The Pass@1 results are shown below. We use the same alpha as in the main experiment.

| Method|HumanEval|HumanEval+|CodeContests|MBPP|
| ------------------------- | ----- | ----- | ----- | ----- |
| GPT4                      | 81.55 | 71.43 | 6.07  | 71.26 |
| MPSC-Uniform              | 89.02 | 75.89 | 7.88  | 74.79 |
| MPSC-MBR                  | 89.02 | 78.66 | 8.48  | 73.24 |
| MPSC-Weighted Cardinality | 89.63 | 78.01 | 10.39 | 74.67 |

The efficacy of MPSC remains evident in its substantial performance gains, even when applied to GPT-4. It should be noted that these gains, while considerable, appear relatively modest compared with the remarkable improvements seen in ChatGPT. This discrepancy is understandable given that MPSC relies on the consistency within a single LLM. Despite ChatGPT and GPT-4 are two versions of the same LLM, subtle variations can still influence outcomes. We posit that employing MPSC with outputs exclusively generated by GPT-4 could potentially unlock unparalleled performance.

Reviewer comment (Reproducibility):
The MPSC is complicate and unstable which limits the reproducibility.

Author response:
We are sorry if our description of MPSC is hard to understand, which may cause your confusion. But we want to emphasize that **MPSC is a simple, intuitive and deterministic algorithm.** 

Firstly, MPSC requires the LLM to generate outputs from different perspectives. And intuitively we can leverage graph to model the relations among them, in which the edge weight w(vi,vj) is well defined as the inter-consistency between two vertices.

With the graph encoding the consistency information, we can then easily employ a simple but effective graph ranking algorithm (i.e. manifold ranking [1]) to aggregate the information and hence select the most consistent solution.

Both the graph construction process (edge weight is obtained through code execution) and the graph ranking process are deterministic. So the reproducibility can be guaranteed.

[1] Learning with Local and Global Consistency

Reviewer comment (Evaluation):
The cost of MPSC is very high which limits its application.

Author response:
First, we want to clarify that sampling 200 solutions for evaluation has been a convention setting in code generation since Codex [3]. And we follow the same temperature setting as in CodeT [4].

To fully address your concern about reproducibility, we resample solutions with 3 random seeds and report the mean Pass@1 performance with std. We only experiment with WizardCoder(7b/13b/34b) on HumanEval here, because the OpenAI API access is both expensive and time-consuming during this period. The results are shown below. It is evident that the improvement $\Delta$ brought by MPSC is quite stable.

|Model Size| Random Selection | MPSC |$\Delta$|
|--|--|--|--|
|7B|52.20(±2.16)|70.97(±0.93)|18.77(±1.74)|
|13B|58.62(±1.22)|76.33(±0.94)|17.71(±2.17)|
|34B|66.04(±1.27)|79.99(±0.72)|13.95(±1.83)|

We are also grateful for your careful consideration about the cost of MPSC in practical scenarios. Therefore, we reduce the solution sampling number into 10 / 50 / 100. The results are shown below. Even with fewer solutions, MPSC can still accurately select the correct ones (if exists) and achieve extraordinary performance.

|#Samples|Method|HumanEval|HumanEval+|MBPP|CodeContests|
|--|--|:--|:--|:--|:--|
|100|random selection|68.38(±0.08)|58.70(±0.12)|66.82(±0.07)|2.51(±0.07)|
|100|MPSC|85.61(±0.3)|76.08(±0.33)|72.19(±0.17)|10.87(±0.29)|
|50|random selection|68.50(±0.24)|58.78(±0.27)|66.85(±0.09)|2.56(±0.06)|
|50|MPSC|85.13(±0.3)|75.75(±0.49)|72.14(±0.36)|10.68(±0.46)|
|10|random selection|68.40(±0.6)|58.85(±0.49)|67.02(±0.12)|2.81(±0.21)|
|10|MPSC|83.42(±0.9)|73.79(±0.53)|72.39(±0.24)|8.73(±1.98)|

[3] Evaluating Large Language Models Trained on Code

[4] CodeT: Code Generation with Generated Tests

Reviewer comment (Theory):
There have been some works investigate the inter-consistency issue [1]. The inter-consistency issue should come from different LLMs other than a single LLM. Therefore, the inconsistency among solutions, specifications, and test cases seems like a kind of intra-consistency within a single LLM.

Author response:
Thanks for your thoughtful advice about the missing related work. 

We need to emphasize that the inter-consistency defined in this work is among different perspectives with respect to one query / question.

We believe that, for a single LLM, there exists a form of consistency in its responses to the same question across different perspectives, which we term inter-consistency. Simultaneously, there is intra-consistency observable in various responses emerging from the same perspective. MPSC is designed to fully exploit both these two kinds of consistency information within one model.

In essence, our emphasis lies in leveraging the inter-consistency of the same LLM. On the contrary, [2] finds the inter-**IN**consistency between different models, and proposes to solve it through a formal debate framework. Even though we happen to use the same name of “inter-consistency”, we refer to different concepts. And their motivation and contribution are quite different from ours.

Thanks for pointing out this paper. We will consider adding it into the related work session in the revision version.

[2] Examining the Inter-Consistency of Large Language Models: An In-depth Analysis via Debate

Reviewer comment (Reproducibility):
MPSC needs to sample 200 solutions for a problem, and a very high temperature should be used to ensure diversity. This raises concerns about its reproducibility and stability. Moreover, it is costly to implement MPSC because 200 solutions must be sampled for each problem.

Author response:
First, we want to clarify that sampling 200 solutions for evaluation has been a convention setting in code generation since Codex [3]. And we follow the same temperature setting as in CodeT [4].

To fully address your concern about reproducibility, we resample solutions with 3 random seeds and report the mean Pass@1 performance with std. We only experiment with WizardCoder(7b/13b/34b) on HumanEval here, because the OpenAI API access is both expensive and time-consuming during this period. The results are shown below. It is evident that the improvement $\Delta$ brought by MPSC is quite stable.

|Model Size| Random Selection | MPSC |$\Delta$|
|--|--|--|--|
|7B|52.20(±2.16)|70.97(±0.93)|18.77(±1.74)|
|13B|58.62(±1.22)|76.33(±0.94)|17.71(±2.17)|
|34B|66.04(±1.27)|79.99(±0.72)|13.95(±1.83)|

We are also grateful for your careful consideration about the cost of MPSC in practical scenarios. Therefore, we reduce the solution sampling number into 10 / 50 / 100. The results are shown below. Even with fewer solutions, MPSC can still accurately select the correct ones (if exists) and achieve extraordinary performance.

|#Samples|Method|HumanEval|HumanEval+|MBPP|CodeContests|
|--|--|:--|:--|:--|:--|
|100|random selection|68.38(±0.08)|58.70(±0.12)|66.82(±0.07)|2.51(±0.07)|
|100|MPSC|85.61(±0.3)|76.08(±0.33)|72.19(±0.17)|10.87(±0.29)|
|50|random selection|68.50(±0.24)|58.78(±0.27)|66.85(±0.09)|2.56(±0.06)|
|50|MPSC|85.13(±0.3)|75.75(±0.49)|72.14(±0.36)|10.68(±0.46)|
|10|random selection|68.40(±0.6)|58.85(±0.49)|67.02(±0.12)|2.81(±0.21)|
|10|MPSC|83.42(±0.9)|73.79(±0.53)|72.39(±0.24)|8.73(±1.98)|

[3] Evaluating Large Language Models Trained on Code

[4] CodeT: Code Generation with Generated Tests

Reviewer comment (Writing):
Understanding the details of the MPSC framework is challenging. A real example of the multipartite graph would be helpful to lower the threshold of understanding.

Author response:
Again, we are sorry for the unclear description of MPSC. Thanks for your advice, we will add a real example of the graph in the revision version.

Reviewer comment (Experiments):
A key issue with the methodology and experiments of this paper is that the proposed method actually requires the execution of code, rather than merely generating it. For instance, in Listing 3 of Appendix 3, one needs to execute the solution to complete the validation with test cases. The authors need to emphasize this distinction from other methods. Clearly, when code execution is allowed, the task becomes easier. This issue makes the main experimental results of the paper, as shown in Table 2, unfair, as other methods do not require code execution.

Author response:
On one hand, we never execute codes with golden test cases, but only with generated test cases. It doesn’t make the task easier because the correctness of generated test cases is not guaranteed. On the other hand, main baselines (including Self-Consistency, MBR-Exec, CodeT) **all require code execution** . Therefore, the comparison in our main experiment is fair.

- Self-Consistency: we follow the setting in [1]. If two solution pass the same set of generated test cases and specifications, we regard them “consistent”. Then we take a majority voting to rank solutions following [2].
- CodeT: CodeT first uses generated test cases to verify each solution by code execution. Then it utilizes RANSAC algorithm to create consensus sets based on execution results. The size of consensus set is then used to rank solutions.
- MBR-Exec: This method ranks solutions by minimum Bayes risk decoding based on the execution results in the generated test cases.

We will make the details of these baselines all clear in our updated paper.

[1] CodeT: Code Generation with Generated Tests  
[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models

Reviewer comment (Experiments):
Is the learning approach designed in this paper overly complicated? Is it possible to avoid using w(vi,vj) and f(vi) and directly employ an MLP or other ensemble methods to obtain the answer? For instance, self-consistency actually uses max-vote directly. Overly complex optimization algorithms make the methodological contributions of this paper ambiguous.

Author response:
We want to argue that the design of MPSC is simple and intuitive.

f(vi) is the ranking score of each output, which is necessary for any ranking algorithm since it is the final objective. The edge weight w(vi,vj) is naturally defined as the inter-consistency of two vertices from different perspectives in the graph, which can be deterministically obtained by code execution. After we build the graph (we will add a real example of the multipartite graph in our paper for clearer understanding), we can easily apply a simple but effective graph ranking algorithm here, i.e. manifold ranking [3], to aggregate the consistency information and select the best solution.

It is clear that **MPSC is not a supervised learning approach**. It’s a non-parametric, deterministic algorithm. We don’t need to involve any learnable parameters, e.g. using MLP. 

There indeed exists other ensemble strategies like self-consistency, and we have included it as baseline. MPSC surpasses it by a large margin.

[3] Learning with Local and Global Consistency

Reviewer comment (Theory):
The specific "perspectives" used in this paper do not entirely align with intuition. For example, in Figure 2, the paper views the solution, specification, and test case as three perspectives, whereas conceptually, they are three answer components.

Author response:
Solution, test case and specification are three important aspects in agile software development methods [4]. They play different roles in the software development lifecycle. They are not different “answer components” because the only answer component required in code generation task is the solution. One can certainly complete this task without introducing test cases and specifications. And our key insight is to make LLM think from these non-essential but significant perspectives, so that we can leverage the consistency information within LLM to further enhance its own capabilities. 

[4] Agile Software Development Methods: Review and Analysis

Reviewer comment (Reproducibility):
The implementation details are unclear, particularly regarding how MBR-Exec is implemented. The reviewer does not understand why MBR-Exec can perform worse than self-consistency. Their understanding is that self-consistency selects the final program based on the exact match, i.e., selecting the most frequently appearing code in all samples, while MBR-Exec selects programs based on the frequency of execution results. They question whether MBR-Exec utilizes the given test cases as in the original paper.

Author response:
>1. How is MBR-Exec implemented? Does MBR-Exec utilize the given test cases as in the original paper?

We are sorry for the missing implementation details of baselines.

For the self-consistency baseline, we actually don’t use the lexical exact match, but follow the setting in [1]. That is, if two solutions pass the same set of generated test cases and specifications, we regard them “consistent”. Then we take a majority voting to rank solutions.

For MBR-exec implementation, we don’t use the golden test cases but the generated test cases for a fair comparison. Because MPSC and other baselines do not execute the golden test cases. 

So it is reasonable that self-consistency exceeds MBR-Exec. We will make these details clearer in the updated paper.

[1] CodeT: Code Generation with Generated Tests

Reviewer comment (Experiments):
Some ablation studies are missing, specifically for MPSC-Label. The reviewer asks how the golden test cases are utilized and whether programs that do not pass the test cases are directly filtered out. They also do not understand why MPSC-Weighted Cardinality can sometimes outperform MPSC-Label.

Author response:
>2. For MPSC-Label, how are the golden test cases utilized? Do you directly filter out those programs that do not pass the test cases?

We are sorry for the misuse of phrase “golden test case”, which may cause your confusion.

To avoid misunderstanding, we want to emphasize that labels used in MPSC-Label setting is the example test cases provided in docstrings (silver test cases), not the test cases used during testing (golden test cases).

We **NEVER** directly filter the programs with silver test cases or golden test cases.

The main idea of MPSC-Label (we refer you to the section 3.3 and Table 1 for more details) is to pass the messages of silver test case vertices to the generated solution vertices through the graph. On the other hand, MPSC-Weighted Cardinality utilizes the intra-consistency within LLM.

They utilize different kinds of information (one from the outsides, one from the insides). The performance is surprising but also reasonable. As shown in [2], the intra-consistency is highly correlated to correctness. Our experiment further demonstrates this point.

[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models

Reviewer comment (Experiments):
Sampling 200 solutions for GPT-3.5 with MPSC, which can outperform GPT-4, is considered very expensive. The reviewer requests results with fewer numbers of samples, e.g., 10 or 100, and asks what pass@200 is, noting that it should be the upper bound of the performance.

Author response:
>3. Do you have results with fewer number of solutions, e.g., 10 or 100 instead of 200? What is pass@200, which should be the upper bound of the performance?

First, we want to clarify that sampling 200 solutions for evaluation has been a convention setting in code generation since Codex [3].

We are grateful for your careful consideration about the costs of MPSC in practical scenarios. Here we add an extensive experiment by only sampling 10 / 50/ 100 solutions (we conduct sampling for 5 times with different seeds, and report mean Pass@1 performance with std error bar). We use the same MPSC-Weighted Cardinality settings as that in the main experiment. It is evident that even with fewer solutions samples, MPSC is able to select the correct ones (if exists), and outperforms baseline (i.e. random selection) by a significant margin.


|#Samples|Method|HumanEval|HumanEval+|MBPP|CodeContests|
|--|--|:--|:--|:--|:--|
|100|random selection|68.38(±0.08)|58.70(±0.12)|66.82(±0.07)|2.51(±0.07)|
|100|MPSC|85.61(±0.3)|76.08(±0.33)|72.19(±0.17)|10.87(±0.29)|
|50|random selection|68.50(±0.24)|58.78(±0.27)|66.85(±0.09)|2.56(±0.06)|
|50|MPSC|85.13(±0.3)|75.75(±0.49)|72.14(±0.36)|10.68(±0.46)|
|10|random selection|68.40(±0.6)|58.85(±0.49)|67.02(±0.12)|2.81(±0.21)|
|10|MPSC|83.42(±0.9)|73.79(±0.53)|72.39(±0.24)|8.73(±1.98)|

Here we also report pass@200, which is the upper bound of all methods. The gap between the upper bound and MPSC is relatively small (8% on HumanEval, 5% on CodeContests, 10% on MBPP), and again proves the effectiveness of our method.
- HumanEval: 93.90
- HumanEval+: 90.24
- CodeContests: 19.39
- MBPP: 83.37

[3] Evaluating Large Language Models Trained on Code

Reviewer comment (Evaluation):
A discussion on the quality of generated test cases and specifications would be helpful, including metrics such as true positive and false negative rates.

Author response:
>4. It is helpful to add discussion on the quality of generated test cases and specifications. 

We have already discussed the quality of generated test cases and specifications in Table 10. Instead of using accuracy, we utilize pass@k to demonstrate the effectiveness of MPSC on ranking other perspectives (i.e. test cases and specifications). And the pass@1 is actually equal to accuracy. So the accuracy of specifications and testcases are as following: 

- HumanEval: Test case 63.83%, Specification 45.66%
- MBPP: Test case 24.54%, Specification 50.83%

It should be noted that the accuracy of specification is not 100% accurate. Because the formal verification of specification requires professional human evaluation, which is both expensive and time-consuming. As an alternative, we apply a looser verification by examining whether a specification can pass golden test cases.

Reviewer comment (Experiments):
The main limitation of the work is the useability on a broader range of tasks. Despite MPSC being claimed to be task-agnostic, only code generation was presented in the paper, which greatly limits the impact of this work. On one hand, the authors only study code competition task, and it is unknown whether the framework can work in code generation in the wild. On the other hand, the authors should consider including at least one more NL task to demonstrate the extensibility of the framework.

Author response:
We have conduct comprehensive experiments on four popular benchmarks of code generation. MPSC has shown superior performance on all four of them and surpassed other methods (even GPT4) by a large margin. To our best knowledge, none of the current LLM-based code generation research, e.g., [1][2][3] conducts experiments of code generation in the wild. And there is also no high-quality such benchmarks. We would be glad to conduct experiments if there is any. In theory, MPSC can be applied for any scenarios in the wild provided that code execution is feasible.

[1] Evaluating Large Language Models Trained on Code  
[2] WizardCoder: Empowering Code Large Language Models with Evol-Instruct  
[3] Code Llama: Open Foundation Models for Code

Reviewer comment (Experiments):
It is unclear whether and how well the framework can generalize towards more perspectives. In code generation, there are only three perspectives, which is quite limited. It would be great to think about and demonstrate that MPSC can work with arbitrary number of perspectives.

Author response:
To avoid potential misunderstanding, we want to underscore that the introduction of “perspectives” is to induce outputs reflecting LLM’s understanding of the query, and therefore better exploit the consistency within LLM. So the number of perspectives should be adapted to the requirements of the given task, rather than being rigidly constrained by an arbitrary number.

Regarding your concern, we would like to discuss about the impact of the number of perspectives on MPSC. It should be noted that the number of perspectives only determines the partitions of the constructed graph. MPSC framework contains two stages: Graph Construction (GC) and Graph Ranking (GR). The GC stage requires to measure pair-wise inter-consistency between vertices from two different perspectives and intra-consistency of each vertex. They both are independent of the number of perspectives. The GR stage is essentially an optimization process over a graph (see Eq.3 for details). Notably, this optimization process, with a closed-form solution (see Appendix A for details), does not leverage the properties of a multi-partite graph and hence remains unaffected by the number of perspectives.

Reviewer comment (Evaluation):
What would be the reason for degradation in MBPP?

Author response:
We conduct a case study about the failure modes of MPSC where the foundation model do generate some correct solutions. We find some serious ambiguities in MBPP because of no example test cases provided in its docstrings. Here we list two typical kinds of ambiguity.

- format ambiguity: The type of return value is not explicitly defined in the prompt. In the following case, LLM generate more than 50 “misguided” answers which return the correct solution but in a `tuple` format.
    - prompt:
        
        ```bash
        def sort_sublists(input_list):
        '''
        Write a function to sort each sublist of strings in a given list of lists.
        '''
        ```
        
    - MPSC selected solution:
        
        ```bash
        for sublist in input_list:
            sublist.sort()
        return input_list
        ```
        
    - golden test cases for testing:
        
        ```bash
        assert sort_sublists((["green", "orange"], ["black", "white"], ["white", "black", "orange"]))==[['green', 'orange'], ['black', 'white'], ['black', 'orange', 'white']]
        ```
        
- semantic ambiguity: The instruction in docstrings is not clear and contains ambiguity. In the following case, LLM generate more than 50 “misguided” answers which regard the `str` variable as a list because the description `given list of words` in docstrings. But it is in fact a string. It also generate more than 250 "misguided" testcases with `str` variable as a `list`.
    - prompt
        
        ```bash
        def long_words(n, str):
        '''
        Write a function to find words that are longer than n characters from a given list of words.
        '''
        ```
        
    - MPSC selected solution:
        
        ```bash
        words = []
        for word in str:
            if len(word) > n:
                words.append(word)
        return words
        ```
        
    - golden solution:
        
        ```bash
        word_len = []
        txt = str.split(" ")
        for x in txt:
            if len(x) > n:
                word_len.append(x)
        return word_len
        ```
        

In both cases, LLM is seriously misguided to a different understanding of docstrings because of the ambiguity. MPSC is designed to fully exploit the consistency within LLM to select the most consistent solution, and hence is seriously influenced. In our opinion, the degradation is not only plausible, but also further justifies the design of our framework.

Reviewer comment (Experiments):
The improvement is diminishing when using a higher k in pass@k. Can the authors perform experiments with a much higher k (e.g. 100) and see the gap there?

Author response:
First we want to emphasize that pass@k with small k (e.g. 1,2,5) is much more important in practical scenario since users won’t expect to do a second selection from a total of 50 solutions. And current works about code generation models (including ChatGPT, GPT4, StarCoder, WizardCoder) only report pass@1 performance.

The diminishing improvement is expected with higher k theoretically. An intuitive way to understand that is from the definition of pass@k, which is the probability of whether the selected $k$ samples out of 200 pass unit tests (Refer to Appendix C for more details). The lowerbound of selection, random selection (also our baseline), has a pass@k value of $1-\frac{\tbinom{200-c}{k}}{\tbinom{200}{k}}$ ($c$ is the number of correct solutions out of 200 solutions). While the upperbound of selection has a pass@k value of 100% (if there exist correct solutions). When k=100, c=6, even the lowerbound can achieve 99%. When k=50, c=10, the lowerbound can achieve 95%. The small gap between lower bound and upper bound makes the evaluation meaningless for all ranking algorithms.

Reviewer comment (Evaluation):
For Bayes Risk, why would using BLEU metrics be preferred especially given the code generation task?

Author response:
We choose BLEU because the MPSC-Bayes Risk setting is designed to measure the **lexical** intra-consistency. And we utilize the BLEU metrics following the first MBR method in NLP [4].

If you refer to the semantical consistency of codes, the structural equivalence (Sec 3.3) reflects some sorts of “MBR” with respect to the code execution results.

[4] Minimum Bayes-Risk Decoding for Statistical Machine Translation

Reviewer comment (Evaluation):
Can the authors discuss the overhead added due to the graph introduced by MPSC?

Author response:
The overhead of MPSC mainly comes from the inter-consistency measurement through code execution. For verification between two vertices, the time is about 1e-4 sec (measured by executing one solution with one test case for 1000 times) in average. Following our main experimental setting (200 solutions & 500 test cases & 100 specifications), the sequential execution time is about $(200\times 500+200\times 100+500\times 100)\times 0.0001s=17s$. It’s worth noting that all the measurement can be executed in parallel. Moreover, as shown in Table 5&6, MPSC maintains the superior performance with only 10% test cases and specifications, which only requires 1.25s with sequential execution.