From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_3qf4
title: Validation vs. verification
comment: Yes, validation is a more appropriate term here than verification.

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Certified (in)correctness and verification vs validation
comment: Thank you for your engagement. Would different word choice change your mind about the paper? We'd like to avoid confusion--and indeed, none of the reviewers seem mislead about what our work actually does--so if different nomenclature is more appropriate, we can straightforwardly swap out certain key technical terms. For instance, the Sketch program synthesizer refers to its bounded checking as "validation" (Solar-Lezama 2008, PhD thesis), which especially in the context of our posted revision's property-base testing phase, could be more appropriate. Another possibility is to *always* refer to these specifications as "certificates of (in)correctness", as the draft already sometimes does, because they can certify the incorrectness of synthesized code by highlighting buggy behavior.

Are either of these changes in word choice acceptable to you?

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Property-based testing and verification
comment: The goal of property based testing is to check if something is true over all inputs. Property based testing, as you note, does not actually check the infinite set of all possible inputs, but instead analyzes the source code of the function to be tested and then constructs a large set of inputs that cover different control flow paths through the program. Our previously reported numbers set the timeout for each example to 200ms and use max 100 examples for each program we test. We can highlight this in a revised appendix.

Would you diagnose the main show-stopper as the use of the word "verify"? We'd like to avoid confusion, and to understand if this is indeed the core issue with the paper--and change the paper's word choice if so, which would be straightforward to do. For instance, the Sketch program synthesizer refers to its bounded checking as "validation" (Solar-Lezama 2008, PhD thesis).

Thanks for engaging with us, and for helping us improve the paper.

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_ZSTh
title: The issues with '''certified correctness'' remains unresolved
comment: I thank the authors for the response. Unfortunately, I remain pretty negative about the paper. For starter, from the authors' response above regarding how language models can generate formal specifications, I guess authors equate "unit tests" with formal specifications, which is obviously wrong. Formal specification are mathematical-based techniques that formally define the semantics of a given program (e.g. pre-, post-conditions), units tests do not fall into this category. Later, authors argue the notion of certified correctness is vague in program synthesis, yes, this is exactly the reason in formal specification, the specification language is never based on natural language but logics/mathematics, so all the claim made in the paper about formal specification, certified correctness should be removed, the contribution of this paper is the incorporation of another input/feature dimension of unit tests for a neural code model which gives better empirical results in certain benchmarks.

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_3qf4
title: Useful updates
comment: Thank you for your detailed response and performing additional experiments to clarify some of the missing aspects.

I like that you have applied property-based testing to the logical specs and discussed this in the paper. I am however confused by your use of term "verification" in the appendix and claim that "all" of the input space is covered by the testing tool. Is this correct? Don't property-based testing tools do random testing (possibly with some feedback)? Do you set a timeout or the maximum number of tests to be generate?

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Summary of changes in response to reviews
comment: Thank you for your time in the review process. The changes you all suggested helped us much improve our manuscript. We would appreciate if you could look at our responses and changes, and let us know if you have further questions. We outline the most important changes below:

1. [Reviewer 3qf4] We reran our system using the same sampling parameters as Chen et al ‘22 (CodeT). This modestly improves our previous quantitative results, and while showing speculyzer’s robustness to these parameters. (It also sets a new state of the art on MBPP.) Please see the new Fig. 3.

2. [Reviewers 3qf4, ZSTh] We added additional discussion of verification, certification, soundness, and completeness (Sec. 2, bottom of pg 3; Appendix A2; A1, “Verification”)

3. [Reviewers 3qf4, ZSTh] To address concerns that the logical relations are not checked over all possible inputs, we ran a property-based-testing checker on the relations, finding that between 40% and 70% of the time, a logical relation might hold over all inputs, not just the inputs Codex proposes. This means that Speculyzer can avoid claiming a spec holds over all inputs and instead provide to the user representative inputs where the relation is true – and conversely, if a spec does hold over every single possible input, we can detect that and output that fact as well, along with inputs that serve as witnesses. Please see the new Table 1 in Appendix A1.

4. [Reviewers myCC, ZSTh] To give insight on how to set the confidence threshold, and to justify our use of a simple probabilistic classifier, we have showed that our model yields well-calibrated probabilistic predictions, which are important for high precision (Appendix A10)

5. [Reviewer 3qf4] We evaluated on the smaller Cushman language model (Appendix Fig. 7-8), finding that our qualitative results do not hinge on using the hefty Davinci model

6. [Reviewer myCC] To understand the results of training on HumanEval and MBPP, we union’d these datasets and computed cross-validated accuracy/trust metrics (Fig. 5). This further confirmed that our results are robust to shifts in the training distribution.

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Author response to ZSTh
comment: Thanks for the feedback! Below we give a point-by-point response:

> how can you make sure the formal specs are generated correctly in the first place

Language models are not perfect at generating specifications, just as they are not perfect at generating programs. But when a language model sufficiently “understands” a problem, our work shows that it can generate high-quality specifications, as well as generating correct solutions. This alignment of programs and specifications signals both program correctness, and also whether the language model can solve the problem in the first place. Our feature space (Equation 1) is designed to detect signals of this spec-program alignment.

>My questions is that how is a language model can be used to train to generate formal specs because obviously there is not enough data of test cases and logic relations given a problem statement in natural language. If authors disagree, then please present your evidence which is not in the paper.

Language models such as Codex are trained on a variety of naturally-occurring source code, which includes unit tests. We believe this is the underlying reason why they are able to generate predicates checking code correctness.

We disagree strongly that the paper does not present evidence of this ability of large language models. Respectfully, we would invite you to re-examine Figures 2 and 6, which give concrete examples of this phenomenon in action. The end of the appendix, referred to on page 9, gives 40 further examples. Were this impossible, it would be hard to explain our quantitative results as well.

> the authors mention ''certified correctness'' or things of that nature. This is obviously problematic... passing test cases, satisfying incomplete or even incorrect logic relations... does not qualify authors to call their approach ''certified correctness''.

Ultimately this program synthesis is happening from natural language: What it means to “certify” against natural language is not obvious, but at minimum must involve translating informal language to formal predicates. To the best of our knowledge our work is the first to explore that idea, but doing so requires stretching the notion of “certificate”. Our notion of “certificate” is similar to unit test harnesses (see pg 1 bullet 2). This differs from other notions of certification, which we clarify by also calling our approach “certificates of (in)correctness” (pg 9), and in the penultimate paragraph of the conclusion.

We believe these are important issues though, so we’ve added an extended discussion to the revision further expanding on these issues (bottom of page 3, bolded section called “Verification” and Appendix A.2).

> Authors need to justify the use of a model for computing the confidence score of each program. All the model does seems to aggregate a bunch of scores w.r.t the metrics presented in Sec 3.2 which is an overkill to me. By the way, how do your threshold defined?

We learn a logistic regressor, which is one of the most basic probabilistic classifiers, and not in our estimation an “overkill”. While one could perhaps dispense with feature engineering given enough data, we're learning our classifier from just a few hundred hundred problems. Because the logistic regressor is trained via maximum likelihood, its predictions tend to be out-of-the-box calibrated, meaning you could set the threshold to 90% and expect ~90% of the programs which pass the threshold to be correct. In the revision we experimentally confirm that the learned classifier is well-calibrated (Appendix A9), which we encourage you to check out.

>The approach... is quite shallow intellectually... it generates programs and predicates independently, and uses predicts to determine the generated code. Personally, I would investigate how to incorporate the feedback... to influence how code is generated on the fly

We would love to explore more complex approaches, such as the one you suggest here or the next steps outlined in our conclusion. However, we were surprised at how well these simple methods worked, and intended this paper partly as a way of advocating for safety and trust in neural code generation, while carving out basic first approaches and metrics. Although the machine learning community has found a resurging interest in program synthesis and has focused on raw accuracy/coverage, we the authors also come from a programming languages background. From that background, trust and assurance are paramount concerns, but it is unclear how to map those on to natural language specifications. We hope that our work can be published in order to outline the shape of the problem, as well as introduce a surprisingly performant basic solution to it, which we hope ultimately spurs more complex developments like those you spell out.

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Author response to 3qf4, part 2 of 2
comment: > What is the oracle model? How do you construct it? What do you mean that CodeT outperforms the oracle?

The oracle model upper bounds the performance of any ranking scheme, given a fixed pool of sampled programs. It measures the performance of a “perfect ranking”, which would always rank a correct program first, provided there is at least one correct program in the pool of samples. In practice, the oracle’s pass@k is just the fraction of programming problems where at least one sampled program is correct.

CodeT, as presented in Chen et al. 2022, used different sampling parameters than we used in our evaluations. (Inala et al. 2022 also uses different sampling parameters from both of us–alas, there is not yet a standard sampling setting). The numbers quoted in Chen et al. 2022 are sometimes higher than our oracle because they found better sampling hyperparameters, giving pools of candidate programs which generally contained more correct sampled programs.

For fair comparison, we also run CodeT on our samples (“CodeT Replication”). By construction this cannot beat the oracle. CodeT Replication is what should be compared head-to-head with our method, but we quoted Chen et al. 2022 for completeness.

As mentioned previously, we are rerunning speculyzer using the exact same hyperparameters that CodeT uses. Because these Codex hyperparameters seem better than the ones we used, we expect that these new results will widen the quantitative advantage of our system over that baseline.

> What is the average number of tests and logical specs respectively sampled per program? What percentage of programs have more than one test and one logical spec? What is the average cluster size?

These statistics are now reported in the new Appendix A.3 of the revision. In short, cluster sizes tend to be on average around 4-5; the number of distinct specifications is around 100 (both for input-outputs and relations); and around 83% of program satisfy at least one input-output, while ~97% satisfy at least one logical relation.

>How is ordinal rank of a feature computed? Is it in comparison to other programs/specs sampled?

Yes, it is in comparison to other programs + specs sampled. We have clarified this in a footnote on page 5 of the posted revision.

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Author response to 3qf4, part 1 of 2
comment: Thank you for the feedback! Below is a point by point response:

> The logical specs are only tested on concrete inputs and are not checked for validity over the input space. Therefore, it does not offer the same trustworthiness as formal verification would.

We agree! (And the last paragraph of the paper outlines similar directions as future work). There are two routes to performing more sophisticated checking of logical relations: 
1. Restrict the output language to a subset of python which can be analyzed by a formal verification tool, following eg [Singh, Gulwani, Solar-Lezama ‘13 PLDI], among others
2. Keep using arbitrary python, but restrict ourselves to more limited forms of checking, such as fuzzing or property-based testing.

Inspired by your comment, we’ve started running QuickCheck-style checking on the logical relations, and using those results as an additional signal of program correctness. We can report back later this week on how much this affects the outputs of the system, which would be incorporated into a final version of the paper.

We also revised the draft to better discuss these issues (pg 3):
> Specifying and certifying the correctness of software is the traditional goal of formal verification methods (Pierce et al., 2022; Baier & Katoen, 2008). We seek trust in slightly different ways: There are no prospects of truly verifying against natural language, so we use less precise, but more human-understandable, kinds of specifications. Rather than specifying exact program semantics in a rich logic, we use unit-test harnesses. A language model generates inputs on which to run those harnesses, instead of verifying across all possible inputs. In principle nothing precludes applying more sophisticated verification techniques to the specifications our system elicits. See Appendix A.2 for further discussion.

The next issue raised is:

> The specs are ranked by satisfaction and selectivity (Section 4.4). The paper does not offer analysis of soundness and completeness of the logical specs. Therefore, whether the specs serve as certificates or not cannot be understood. 

Thanks for raising these issues. We have added the following to the revision (appendix A2, referred to above):
> The formal methods community has traditionally sought methods that are sound and complete (Baier & Katoen 2008). Soundness means that whenever a program satisfies a spec,
we identify it as such. Completeness means that every spec we identify as valid for a
program is indeed entailed by that program. Generally, speculyzer’s verification step 
overapproximates the set of specs a program satisfies, achieving completeness at the expense of soundness. This overapproximation only occurs for logical relations: we’re sound
and complete for input-outputs, because they assert a property of the program only on a
single input. But because the language model constructs specific inputs on which to run the
logical relations, we cannot assert that they hold generically for every single possible input.
The practical impact of this unsoundness for logical relations is that a human user has to
inspect the candidate inputs on which the language model probes the relation. In practice,
we find that the neural network generates fairly representative inputs (Appendix A.11),
something that AlphaCode (Li et al., 2022) also found.
> In principle, nothing precludes running a model checker or solver to check that the logical
relations hold over every single possible input, which would make the method sound and
complete w.r.t. the programs and specifications. However, this would not eliminate the need
for a human to examine the AI-generated specifications: ultimately, the true specification is
in natural language, and there is no sound and complete way of verifying against an informal
language.

Next, regarding the contributions of the paper:
> the paper's main contribution is suggesting a ranking technique.

As noted by Reviewer myCC, we contribute a much broader toolkit for building trust into natural-language based program synthesizers. This is more than just ranking programs, and includes predicting program correctness (is this program actually correct, and not just ranked higher than another program); favoring precision over recall; and synthesizing specifications that efficiently communicate the behavior of the synthesized code.

> The paper uses its own sampling parameters to regenerate CodeT's results (Section 4.1). How sensitive are the results to this? Can you report the results obtained using CodeT's sampling parameters?

Based on early experiments, the relative performance of different models remains stable across sampling parameters. However, the exact performance of a ranking algorithm depends on what is being ranked, which depends on the sampling parameters. We are running our system using CodeT’s sampling parameters, and will report back later this week.

From: ICLR.cc/2023/Conference/Paper4076/Authors
title: Author response to myCC
comment: Thanks for the feedback! Below we give a point-by-point response.

> The authors state they use different prompts to generate S (given an initial prompt to sample p from P^{LM}( | prompt)). Taking Figure 2 as an example, how can the LLM generate P and R given the same prompt?

Thank you for inviting this clarification. We automatically transform the programming problem prompt (Figure 2, upper left, “Generating Programs”) to generate each of the two kinds of specifications. One of those prompt transformations elicits input-output specifications (Figure 2, upper right, “Generating Input-Output Specifications”), while the other elicits logical relations (Figure 2, bottom, “Generating Logical-Relation Specifications”). Although Figure 2 showed examples of the prompt transformations, the paper would be improved if the manuscript precisely specified these transformations, instead of just giving examples. The revision further describes the transformations in Appendix A10, which the main text refers to on pg 4.

> It seems that speculyzer i/o only and rels only perform similarly (as well as to "our" too). As such, it seems they don't provide any stronger outcome when it comes to evaluating pass@k… [also] CodeT seems to perform very much in line (sometimes better than) speculyzer

The full speculyzer system achieves the highest pass@k when k=1, which is the most realistic k value: The value of k corresponds to the number of programs (and specs) that a human user has to manually inspect. As k becomes large, the difference between systems washes out. (And despite their enthusiastic criticism, Reviewer ZSTh is correct when they say speculyzer “outperforms the latest advances in program synthesis”, at least on these datasets.)

To best understand the CodeT results, it’s helpful to note that we compare against two CodeT pass@k numbers:

1. CodeT Replication, which runs CodeT on the same data that speculyzer runs on. This means we sampled programs and input-output test cases using Codex sampling parameters identical to the ones speculyzer used, and then reranked those samples using CodeT’s ranking algorithm

2. CodeT Chen et al 2022, which reports verbatim pass@k numbers from CodeT’s paper. These numbers come from running CodeT’s ranking algorithm on different data, ie with different Codex sampling parameters.

CodeT Replication is the relevant comparison, because pass@k measures how good a ranking algorithm is, which is going to depend on which samples are being ranked.

Speculyzer outperforms “CodeT Replication” across the most important accuracy and trustworthiness metrics: pass@1, AUC, recall@high precision, and F1 score. Speculyzer is not as good as “CodeT Chen et al 2022” on MBPP pass@1 (by 1%), but Speculyzer is better on HumanEval pass@1 (by 12%). Distinctions between approaches wash out at k=10, as expected.

Thank you for noting a further qualitative advantage of Speculyzer: it generates a wider variety of specifications (ie logical relations). It also ranks the specifications, effectively synthesizing the best specs for each program (Section 4.4), which is important for communicating the (in)correctness of a program.

We also present new analyses showing that Speculyzer gives much better calibrated predictions, meaning that when it predicts with high certainty that a program is correct, that program actually is much more likely to be correct. This is important for high-precision prediction. We would be very appreciative if you would check out the new Appendix A9 / Figure 9 for these new calibration results.

> what would be the results if speculyzer were trained on HumanEval and MBPP?

This is an interesting question, and we are running the experiment right now. We can update you later this week.

> How do you learn the threshold for the confidence score? Is it just a threshold on the PR curve?

Ultimately the threshold depends on the level of precision required, and is indeed just a threshold on the PR curve. But because we learn a probabilistic classifier via maximum likelihood, speculyzer is out-of-the-box “calibrated”, meaning you could set the threshold to 90% and expect ~90% of the programs which pass the threshold to be correct.

As noted, the new Fig 9 and Appendix A9 of the revision analyze speculyzer’s calibration, which we would encourage you to check out.

> Where do you get the ground-truth testcases G? (How is it to get them and how scalable is the whole process?)

Ground truth test cases are used to train the system and compute statistics on how accurate+trustworthy the system is, but would not be used in a hypothetical deployment of the system. Our ground truth test cases are provided as part of the HumanEval and MBPP datasets.

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_myCC
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors explore how LLMs can help towards achieving program
synthesis tasks that provide trustworthiness guarantees. To this end,
they do sample synthetized programs conditioned on a natural language
prompt as well as specifications obtained by transforming the prompt
properly. The authors' system focuses on returning programs that
satisfy specs (input-output and logical relations) prioritizing
Precision over Recall.
strength_and_weaknesses: I applaud the authors for motivating their work so well and providing
code and data to reproduce their approach. The limits of LLMs in
generating bug-free code have been explored recently [1] and the need to
build trust in next-generation neural synthesizers is real.  I like
the approach of building trust in such systems by prioritizing Precision
vs Recall, and providing a certificate of correctness built around unit
testing with human-interpretable guarantees.
clarity,_quality,_novelty_and_reproducibility: The authors provide code and data to help reproduce their work. 
summary_of_the_review: There are some aspects of the methodology I could not grasp from the
paper. The authors state they use different prompts to generate S (given
an initial prompt to sample p from P^{LM}( | prompt)). Taking Figure 2
as an example, how can the LLM generate P and R given the same prompt?
Section 3.1 states the prompt is "deterministically transformed" (as
shown in Fig. 2 but it's unclear what process one should follow and how
challenging it is to apply such changes. This is key as the whole
approach builds on this aspect. (I understand the authors rely on the
HumanEval dataset as well, which includes at times input-output specs,
but this serves as an evaluation rather than representing a realistic
setting.)

It seems that speculyzer i/o only and rels only perform similarly (as
well as to "our" too). As such, it seems they don't provide any stronger
outcome when it comes to evaluating pass@k. However, what are the
implications of trust? Are rels more "convincing" than i/o? Are rels
easier to obtain (in real settings) than i/o? This would help
understand the trustworthiness level of the approach in real
settings. (This is covered somehow a little in Section 4.2 but I wonder
whether the authors could expand on it.)

One concern I have is on the results; CodeT seems to perform very much
in line (sometimes better than) speculyzer (e.g., see Fig. 3). Why
should one choose speculyzer over CodeT? Is it because of the additional
guarantees of correctness?

With respect to the experiments outlined Section 4.3 (and Figure 5),
what would be the results if speculyzer were trained on HumanEval and
MBPP? I would assume this would generalize even better. Perhaps, this
offers some opportunity to discuss sampling bias further.

Additional questions:

How do you learn the threshold for the confidence score? Is it just a
threshold on the PR curve? Where do you get the ground-truth testcases
G? (How is it to get them and how scalable is the whole process?)

[1] Pearce et al. Pop Quiz! Can a Large Language Model Help With Reverse
Engineering? ArXiv 2022. https://arxiv.org/abs/2202.01142
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_ZSTh
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper proposes a program synthesis approach that takes into account the synthesis of formal specs in addition to the generation of source code. The benefit of the approach is that generated code can be checked against the specs so that the correctness of the code can be verified before hand. The paper presents an evaluation of the approach and results show the new approach outperforms the latest advances in program synthesis.
strength_and_weaknesses: Pros

* Adding the dimension of generating formal specs is an interesting idea.

Cons
* I was very curious to learn the part for generating formal specs, and left very disappointed and confused in the end how that is done specifically. The only place I can find related is in Sec 3.1 ''*Both sets are sampled
using a pretrained language model, which can probabilistically generate program source code conditioned on a prompt.*''. My questions is that how is a language model can be used to train to generate formal specs because obviously there is not enough data of test cases and logic relations given a problem statement in natural language. If authors disagree, then please present your evidence which is not in the paper.

-  Still related to the generation of formal specs, how can you make sure the formal specs are generated correctly in the first place. If the specs are wrong, then very likely the synthesis of source code will be severely affected because not satisfying a wrong spec does not mean the program is incorrect. Given the way the confidence score of each program is computed, you would probably unfairly penalize correct programs.
 
* In various places of the paper, the authors mention ''**certified correctness**'' or things of that nature. This is obviously problematic as ''**certified correctness**'' refers to theoretic guarantees of program's run-time behavior. There is a whole field called static analysis/program verification dedicated to that. The notion of correctness used in this paper (*e.g.,* passing test cases, satisfying incomplete or even incorrect logic relations) does not qualify authors to call their approach ''**certified correctness**''.

-  Authors need to justify the use of a model for computing the confidence score of each program. All the model does seems to aggregate a bunch of scores *w.r.t* the metrics presented in Sec 3.2 which is an overkill to me. By the way, how do your threshold defined?

* The approach, albeit may have merit, is quite shallow intellectually. In essence, it generates programs and predicates independently, and uses predicts to determine the generated code. Personally, I would investigate how to incorporate the feedback (e.g. which predicts are satisfied or violated, how are they satisfied or violated, what could have been the causes) to influence how code is generated on the fly such that the final program could satisfy as many predicates as possible if not all.


clarity,_quality,_novelty_and_reproducibility: The paper presents a novel idea, however, it lacks the necessary details. I don't think reproducibility matters in this case as the paper as it stands is far from acceptable.
summary_of_the_review: See above
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper4076/Reviewer_3qf4
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper postulates that the trustworthiness of a program output by a language model can be improved by sampling predicates and test cases for the sampled program. For this, they design prompts to separately sample unit tests and test harnesses. The test harnesses are expected to encode logical constraints on input-output and are accompanied by a concrete test case. They train a linear regression model on features of the pass/fail matrix of specs against programs and use it to rank the candidate programs. An evaluation on HumanEval and MBPP with Codex is conducted to demonstrate the effectiveness of the method against a closely related work CodeT, and cluster and random ranking.
strength_and_weaknesses: Strength
---------
1. An interesting take on using a language model to generate specs, both logical and tests.
2. Using the satisfaction of programs and specs to rank the candidate programs and specs satisfied by them.

Weakness
---------
1. The logical specs are only tested on concrete inputs and are not checked for validity over the input space. Therefore, it does not offer the same trustworthiness as formal verification would.
2. The specs are ranked by satisfaction and selectivity (Section 4.4). The paper does not offer analysis of soundness and completeness of the logical specs. Therefore, whether the specs serve as certificates or not cannot be understood. In the absence of this, the paper's main contribution is suggesting a ranking technique.
3. The paper uses its own sampling parameters to regenerate CodeT's results (Section 4.1). How sensitive are the results to this? Can you report the results obtained using CodeT's sampling parameters?
4. The paper evaluates only one generative model and its applicability to other models is not demonstrated.
5. Some aspects, listed in the next section, need clarification.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly easy to follow and the authors state that they have uploaded a reproducible package.

I request the authors to clarify the following:
1. How is ordinal rank of a feature computed? Is it in comparison to other programs/specs sampled?
2. What is the average number of tests and logical specs respectively sampled per program? What percentage of programs have more than one test and one logical spec? What is the average cluster size?
3. What is the oracle model? How do you construct it? What do you mean that CodeT outperforms the oracle?
summary_of_the_review: The paper has an interesting take on using a language model to supply logical specs and tests along with programs, and using them to improve trustworthiness/ranking. The soundness/completeness of specs and applicability to models besides Codex are not explored, and how the paper will work under different sampling parameters needs clarification.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold