Paper: Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain (ICLR-2024)

Reviewer comment (Evaluation):
The IdentityChain framework assumes that the underlying code model performs well at both tasks (NL-to-PL and PL-to-NL). It is possible that the model is good at one and not the other, and it is not possible to decouple these in their framework.

Author response:
1. We strongly agree that *"It is possible that the model is good at one and not the other"* and it is important to *"decouple these in their framework"*. In **Section 4.3**, we expressed our intention to build a holistic framework that evaluates the NL-to-PL Accuracy, PL-to-NL Accuracy, and Self-Consistency of a model separately at the same time. 

2. Particularly, the metric $SSC_n$ which measures Strong Self-Consistency gives the model developers a summary of the model's overall performance. Then they can examine $Pass@1$ which measures NL-to-PL Accuracy, $SC_1$ which measures PL-to-NL Accuracy (explained in **Section 4.3, Paragraph 3**), $SC_n$ which measures Self-Consistney to determine which aspect is the weakest.

3. Following this line of reasoning, we evaluated these 4 aspects of the models and showed the results in **Table 1**. Comparing the $Pass@1$ score and the $SC_1$ score, we can tell whether a model is worse at NL-to-PL or PL-to-NL. For example, for **GPT-4**, the PL-to-NL Accuracy is higher. For **Code Llama-Instruct-7B**, they are about the same. For **GPT-3.5**, the PL-to-NL Accuracy is significantly lower.

Thank you for the reply. We would like to further clarify on **Weakness 1**.

For the statement *"For GPT-3.5, the PL-to-NL Accuracy is significantly lower"*, we agree that the performance is not consistent across MBPP and HumanEvalPlus. However, we mainly referred to the results of HumanEvalPlus because it is a more accurate benchmark with **16.5** test cases per task, while MBPP only has **3** test cases per task (**Section 5**). This is also the reason why we chose HumanEvalPlus instead of the original HumanEval since the test case coverage is significantly enhanced.

For the comparison between $Pass@1$ and $SC_1$, in our framework, they are both measuring accuracy (**Section 4.1, Section 6.4**). In a chain $nl_0 \to pl_0 \to nl_1 \to ...$, the $Pass@1$ score calculates the percentage of $pl_0$ that passes all test cases, which is NL-to-PL Accuracy. As defined in **Section 3.2**, the $SC_1$ score calculates the percentage of $nl_1$ that is semantically consistent with $pl_0$, which is PL-to-NL Accuracy.

Indeed, these two scores separately measure the model's ability of NL-to-PL and PL-to-NL Generation, and they won’t be comparable if they are not both measuring accuracy. However, in our framework, $Pass@1$ and $SC_1$ are both measuring accuracy, which are absolute measurements *i.e.* how far away the model’s current performance is from absolute correctness. In this sense, we can compare them to see if the model is further from absolute correctness regarding NL-to-PL Generation or PL-to-NL Generation.

Currently, no metric can directly measure the semantic consistency between $nl_1$ and $pl_0$. Therefore, as discussed in **Section 4.1**, we design a relaxed accuracy metric **TOM** score as an implementation of the ideal $SC_1$ score and show in **Section 6.2** that **TOM** score correlates better with ground truth than other existing metrics. We recognize that there will be better metrics than **TOM** score in the future, but within our framework, the definition of $SC_1$ is separated from the implementation. Therefore, future work can simply replace **TOM** score with a better metric as an improvement of the IdentityChain framework.

Reviewer comment (Evaluation):
Based on my previous two points, it seems that the code summarization task (PL-to-NL) may not be symmetric to the (NL-to-PL) task. Namely, in $p_i \rightarrow n_{i+1}$, the model could generate a high-level natural language summary rather than a holistic description of $p_{i}$ with sufficient details for generating $n_{i+1} \rightarrow p_{i+1}$, with the expectation that $p_{i+1}$ would be semantically equivalent to $p_{i}$. It is likely that the pretraining data for these code models include a lot of natural language with high-level descriptions of accompanying code (e.g., docstrings, NL StackOverflow text near code snippets). If this is the case, then the model is likely not great at generating holistic natural language descriptions for code. Therefore, it is possible that the model is very good at NL-to-PL (as per the high accuracy scores) but not as good at PL-to-NL, which results in low self-consistency scores. However, since it is not clear how to decouple this in the IdentityChain, it is difficult to clearly see this.

Author response:
To conclude, in our paper, **the models are tasked and enforced to generate detailed natural language specifications.** In this setting, our results showed that some models are worse at PL-to-NL Generation while some are worse at NL-to-PL Generation.

Reviewer comment (Evaluation):
We could construct a perfectly self-consistent and not-so-useful CLM. For example, a CLM could generate summarization nl_1 exactly the same as pl_0. It is likely that the model generates pl_1 semantically equivalent to the input pl_0 in the next step. The chain goes on and the model is mostly perfectly self-consistent. Based on that, we could have very high self-consistency but that does not say much about the model.

Author response:
1. We would appreciate a clarification on "*CLM could generate summarization nl_1 exactly the same as pl_0*". Does it mean that the model copy-pastes $pl_0$ as $nl_1$? If so, this model is not performing PL-to-NL Generation and therefore out of the scope of self-consistency evaluation by our definition. For our experiments, we carefully designed our [prompt template (run_identity_chain_openai.py, line 61)](https://github.com/anonymousauthor567/IdentityChain/blob/main/examples/run_identity_chain_openai.py) including a one-shot example to make sure that the model actually generates a natural language specification.

2. We agree that *"We could construct a perfectly self-consistent and not-so-useful CLM."*. In **Section 4.3, Paragraph 4**, we mentioned that *"An ideal model should be both accurate and self-consistent.
An accurate but not self-consistent model is not trustworthy, while a self-consistent but not accurate model is useless."* This is particularly why we defined the concept of **Strong Self-Consistency** in **Section 3.1, Definition 2**. An ideal model should be strong self-consistent, which is both accurate and self-consistent.

3. More importantly, we discuss in **Section 4.3** and show through results in **Table 1** that IdentityChain is a holistic evaluation framework that efficiently evaluates NL-to-PL Accuracy, PL-to-NL Accuracy, and Self-Consistency at the same time. In **Section 6.4**, we further discuss that model developers can use IdentityChain to pinpoint the weaknesses of their models.

Reviewer comment (Experiments):
An important aspect of consistency is the consistent behavior to transformations of the prompts with equivalent semantics. This paper does not touch upon it.

Author response:
1. We agree that *"An important aspect of consistency is the consistent behavior to transformations of the prompts with equivalent semantics."* We highlight in **Section 1** of the updated version that **"Self-Consistency is different from Consistency"**. We also discussed in the related work **Section 2, Paragraph 2** that conventional consistency violation is different from self-consistency violations.

2. We believe that self-consistency is a different property of a trustworthy model since self-consistency violations reveal that **the model doesn't even understand its own output**. However, conventional consistency evaluation settings involve human-crafted transformations. It is to some extent acceptable if a model doesn't understand some human-crafted transformations since it might have not seen similar training data, but it's a more serious issue that the model doesn't understand itself.

3. More importantly, in an IdentityChain $nl_0 \to pl_0 \to nl_1 \to ...$ as illustrated in **Figure 1**, $nl_1$ is in fact a **semantic-preserving transformation** of $nl_0$, which is difficult to obtain using rule-based human-crafted transformations.

Reviewer comment (Experiments):
While the concept and the proposed framework sound general to any task, it seems to only apply to code synthesis and summarization. What about other tasks for code such as reasoning about code, fixing bugs?

Author response:
1. Thank you for asking! We added a Future Work Section in **Appendix A**, mentioning that it's easy to incorporate PL-to-PL and NL-to-NL Generation into our framework, and we plan to do that as an immediate next step.

2. For reasoning tasks like Vulnerability Detection, conceptually they can be evaluated together with PL-to-PL Generation following the setups in this [related work](https://arxiv.org/abs/2305.11662), which asks a model to translate a QA task from English to another language and ask the same model to answer the QA task in both languages.

3. We recognize that our current definition of self-consistency requires tasks with a symmetric nature, but the definitions can be extended to further incorporate tasks with semantic entailment relationships. For example, if a model is capable of both Bug Detection and Bug Fixing, then it should be self-consistency across detection and fixing.

Reviewer comment (Writing):
At the end of Page 2, it is said that “the underlying semantics of pl1, nl1, pl0 should all be the same. We call such a behavioral property self-consistency”, but in the next paragraph, the authors write “a model’s accuracy is low, its self-consistency can remain high when the model, despite making initial errors, consistently maintains those errors”. Please clarify.

Author response:
1. By *“the underlying semantics of pl1, nl1, pl0 should all be the same. We call such a behavioral property self-consistency”*, we meant that **Self-Consistency** doesn't require the initial generation from $nl_0$ to $pl_0$ to be accurate. As long as the semantics of $pl_0$, $nl_1$, and $pl_1$ are the same, the model is self-consistent.

2. In contrast, **Strong Self-Consistency** requires the semantics of $nl_0$ to be the same as $pl_0$, in addition to being self-consistent. Therefore, we wrote that *“a model’s accuracy is low, its self-consistency can remain high when the model, despite making initial errors, consistently maintains those errors”.*

3. **"Self-Consistency is different from Accuracy"** is one of the key insights that we want to provide to the readers. Ultimately, we want a model to be both self-consistency and accurate, as elaborated in our response to Weakness 1.

Reviewer comment (Writing):
“it relatively improves GPT-3.5 by 57.5% in FPS-5” What is FPS-5 here?

Author response:
1. Thank you for pointing that out. It is fixed in the updated version.