I Speak, You Verify: Toward Trustworthy Neural Program Synthesis
Anonymous authors Paper under double-blind review
Abstract
We develop an approach for improving the trustworthiness and overall accuracy of program synthesizers based on large language models for source code.
Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave.
We learn to analyze the agreement between programs and predicates to judge both which program is most likely to be correct, and also judge whether the language model is able to solve the programming problem in the first place.
This latter capacity allows favoring high precision over broad recall: fostering trust by only proposing a program when the system is certain that it is correct
The importance of trust
Picture a future where AI systems attempt to close GitHub issues by generating source code given only the natural language of the GitHub issue.
Such systems might not come out next year, but when or if they ever do, they will likely leverage large neural network language models for source code (Chen et al., 2021; Austin et al., 2021).
These neural systems are good, but not perfect.
Suppose 75% of the time, such systems propose a correct fix to the
GitHub issue.
The other 25% of the time, they produce plausible looking code containing subtle bugs.
Would you use this system?
Most engineers would be reluctant to use such a system, because it fails to build trust with the user.
When it fails, it cannot detect its own failure.
When it succeeds, it cannot construct a human-comprehensible certificate of its success.
In this paper we seek steps towards rectifying this lack of trust.
Concretely, our goal is to build natural-language conditioned neural program synthesizers that are more trustworthy along several dimensions:
• We want systems that, when they cannot solve a programming problem, simply return no answer, rather than return a (possibly subtly) incorrect program.
We
conjecture that it is better to fall back on the human programmer, rather than risk introducing bugs.
Contrast the situation with natural language translation: Unlike natural language, programs are brittle, and so must be exactly correct.
And debugging bad code, unlike proofreading language, can be more difficult then just writing it yourself.
• We want systems that can produce a human-understandable certificate of the correctness of the synthesized code.
This activity is common among human engineers, who often write test harnesses for new code.
Similarly, our system proposes predicates testing its solutions, which act as a human-comprehensible signal of the code’s (in)correctness.
• Ideally, trustworthy systems should be more accurate overall, solving more programming problems.
This goal would seem to be in tension with the previous two.
Surprisingly we find our methods for building trust also serve to boost overall accuracy on natural language to code generation problems as well.
Our high-level approach has a neural network propose candidate program solutions and independently propose predicates that correct solutions should satisfy, known as specifications (‘specs’, Fig. 1).
We consider two kinds of specs: (1) input-output test cases, and (2) test harnesses specifying logical relations between inputs and outputs, also known as functional specifications (Solar-Lezama, 2008).
In general, a spec can be any mechanically checkable property.
We check the programs against the specs, and learn to use this checking to predict if the system knows how to solve the problem at all, and if so, which program(s) are probably the right solution.
Intuitively, we ask the language model to ‘check its work’ by generating specs.
We call our approach speculyzer , short for ‘Specification Synthesizer’, because in addition to synthesizing programs, it synthesizes specs.
WITH
[Figure]
Related Work
Program synthesis.
Automatically constructing software has been a longstanding goal of computer science (Manna & Waldinger, 1979; Gulwani et al., 2017).
Classic program synthesizers input a formal specification of what the program should do, and then either search or logically derive a program guaranteed to satisfy that formal specification (Alur et al., 2013).
This formal specification could come from a rich, expressive logic (e.g. Polikarpova et al.
(2016)) or less precise, but still formal modalities such as input-output examples (Gulwani, 2011).
Classic program synthesizers assume it is possible to verify the correctness of a candidate program.
This verification assumption allows a generate-and-test approach to always eventually find a satisfying program, though practical program synthesizers are more clever in how they search for programs (Solar-Lezama, 2008), including incorporating guidance from neural networks (Chaudhuri et al., 2021;
Ellis et al., 2021).
Large language models for source code.
Our work uses large language models for source code (Chen et al., 2021;
Austin et al., 2021).
These neural networks generate source code conditioned or ‘prompted’ by a mix of code and natural language (the natural language is usually represented as code comments).
Such language models are typically implemented as very large transformers (Vaswani et al., 2017; Brown et al., 2020).
Following the introduction of large transformer-based language models for source code, there has been work on how to boost the accuracy of those models.
Here, accuracy means the probability of sampling a correct program conditioned on a natural-language prompt.
Accuracy is often measured by functional correctness with the pass@k metric, which considers drawing k IID samples from the language model and testing if any of those samples pass a set of holdout test cases.
Toward boosting pass@k , researchers have considered clustering sampled programs according to the outputs they produce on test inputs (Shi et al., 2022;
Li et al., 2022).
For example, AlphaCode prioritizes large ‘clusters’ of samples with the exact same input-output behavior (Li et al., 2022), effectively reranking the samples from the language model according to how likely they are to solve the task.
A complementary reranking strategy is to train a second neural network to predict program correctness, as explored in Inala et al.
Another approach is to ask the language model to ‘show its work’ by prompting it to generate/use intermediate evaluation states, known as ‘chainof-thought-prompting’ (Wei et al., 2022) and ‘scratch pads’ (Nye et al., 2021).
Our goal of having our model not make predictions when it doesn’t think it can get it right is related to the recent Grammformer introduced by Guo et al.
(2022), which is a Codex-like model outputting regex patterns containing wildcards where the model is uncertain.
The closest work to ours is the concurrently developed CodeT system (Chen et al., 2022).
CodeT independently proposed generating programs as well as input-output test cases, with the goal of boosting pass@k .
The qualitative difference between our systems is that we designed speculyzer to build trust by synthesizing specifications–only boosting pass@k as a side effect–and incorporated input-output test cases as a special case of specs in general.
Engineering safe, trustworthy language models has received considerable attention by the
AI safety (Thoppilan et al., 2022) and AI alignment communities (Kadavath et al., 2022).
These works find that one can train classifiers which predict the truthfulness or safety of language outputs by inspecting the hidden activations of the model or even by simply ‘asking’ the model if its output is correct or safe.
We see this family of efforts as complementary: For programs, it is possible to formally specify correctness properties, which is not generally true in NLP, so we focus on formal properties (specifications) here.
Nonetheless, one can train statistical predictors of program correctness (Inala et al., 2022), and in fact these synergize with formal notions of correctness (Chen et al., 2022).
Broadly however, we think that program synthesis offers unique opportunities for building trust through symbolic methods.
Although statistically reranking language model outputs via a second neural network improves raw performance, we believe it is a suboptimal trust-builder: an inscrutable neural network cannot guarantee the correctness of another inscrutable network.
Here we advocate that properties which are symbolically verifiable and human-comprehensible should play a role, and examine certain specifications as basic examples of such properties.
Verification.
Specifying and certifying the correctness of software is the traditional goal of formal verification methods (Pierce et al., 2022; Baier & Katoen, 2008).
We seek trust in slightly different ways: There are no prospects of truly verifying against natural language, so we use less precise, but more human-understandable, kinds of specifications.
Rather than specifying exact program semantics in a rich logic, we use unit-test harnesses.
A language model generates inputs on which to run those harnesses, instead of verifying across all possible inputs.
In principle nothing precludes applying more sophisticated verification techniques to the specifications our system elicits.
See Appendix A.2 for further discussion.
Methods
Given a natural-language prompt describing a programming problem, our goal is to construct a ranked list of candidate program solutions, or to output an empty list whenever the system cannot solve the programming problem.
Our approach independently samples a set of candidate programs P and a set of candidate specs S .
Specs can be either input-output testcases, or logical relations (Fig. 1).
We write T by confidence score and returned to the user as possible solutions, together with certain specs they pass.
Returning specifications allows the user to verify that the code has the intended behavior.
This architecture lets the system learn how to predict when it cannot solve a problem, and also learn to rank candidate solutions and their corresponding specs.
3. Sampling programs and tests
Given a string prompt describing a programming problem, we sample n = 100 candidate programs (the set P ) and candidate specs (the set S ).
Both sets are sampled using a pretrained language model, which can probabilistically generate program source code conditioned on a prompt.
We write P LM ( ·| prompt ) for the conditional distribution over programs, given prompt .
If a program p ∈ P , then p ∼ P LM ( ·| prompt ).
To sample specs, we deterministically transform the prompt as in Fig.
2 and Appendix A.10, then draw iid samples from the language model to construct relations R and input-output test cases T .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
correctly.
[Figure]
"" output list = sub list(nums1, nums2) # check if the length of the output list is the same as the lengths of the input lists assert len(output list)
== len(nums1)
== len(nums2)
# check if the output list has the expected elements for i in range(len(output list)): assert output list[i]
==
nums1[i] - nums2[i] # run the testing function ‘test sub list‘ on a new testcase test sub list([1, 2, 3, 4], [10, 9, 8, 7])
3. Scoring and analyzing test coverage
Given programs P and specs S , we compute a confidence score for each p ∈ P measuring how likely p is correct.
Assuming, on average, specs correctly formalize the informal naturallanguage intention, satisfying more specs should increase our confidence in a program.
Additionally, if many sampled programs exhibit identical behavior on the specs, then we should increase our confidence in those programs, because this indicates high marginal probability of that behavior under P LM ( ·| prompt ).
This ‘clustering’ of candidate solutions according to their execution behavior, and prioritizing large clusters, has been successfully used by AlphaCode (Li et al., 2022), Minerva (Lewkowycz et al., 2022), and others (Shi et al., 2022).
It is also related to observational equivalence from classic program synthesis (Udupa et al., 2013), which treats programs as identical if they have the same outputs on test inputs.
While cluster size and spec pass rate give confidence in individual programs, global features of the distribution of sampled programs can indicate whether this system might be able to solve the problem in the first place.
So, we also compute the entropy over cluster assignments: diffuse clusterings could suggest lack of confidence.
Finally, we compute a real-valued score for each program p ∈ P using a logistic regressor over features ϕ ( p, P , S ).
The features ϕ ( p, P , S ) include testcase pass rate (fraction of inputoutput specifications passed), relation pass rate (fraction of logical-relation test-harness specifications passed), cluster size (fraction of other programs with the same behavior on the specifications), the ordinal rank 2 and logarithms of the preceding features, and the entropy of cluster assignment distributions for both input-output and logical-relation specifications, for a total of 18 features.
We clarify the meaning of those features below: score
[Equation]
learned θ , features ϕ
Components of ϕ ( p, P , S ), plus logs and ordinal ranks:
[Equation]
p satisfies
[Equation]
relationPass
[Equation]
clusterSize( p, P , S ) =
X p ′
[Equation]
Y
[Equation] [Equation]
p
[Equation]
# progs.
w/ same spec behavior clusterEntropy( p, P , S ) =
H [ A ] cluster assignment A : S → { 0 , 1 } where P [ A ] ∝ X p ′ ∈P
Y s ∈S 1 [( p ⊢ s ) =
A ( s )] cluster assignment distribution
We fit θ via maximum likelihood on a corpus D containing triples ⟨P , S , G⟩ of programs P and specifications S , both sampled from the same prompt, and ground-truth testcases G , which serve as a proxy for program correctness.
The ground-truth testcases G are assumed to be unavailable at test time, because our goal is synthesis from informal specifications like natural language.
We use gradient ascent to maximize the log likelihood, L :
[Equation]
where σ ( · ) is the logistic sigmoid function.
3. Test time metrics
Precision-Recall.
Ultimately our goal is a trustworthy system that proposes program solutions whenever it can, but avoids proposing buggy code.
Toward those ends, we seek high precision without sacrificing recall .
High precision means that when the system suggests a program, it is probably correct.
Precise systems foster trust because they don’t propose wrong answers, though they may decline to provide an answer in the first place.
High recall means a correct program achieves the top rank:
In other words, the system can solve a lot of programming problems, though it might make more mistakes in the process.
The tradeoff between precision and recall can be tuned by a thresholding parameter, τ .
A candidate program is discarded if its score falls below the threshold τ .
If all programs are discarded, the system declines to provide an output for the programming problem, and otherwise the system outputs a ranked list of programs sorted by score.
We define Precision@ k and Recall@ k , which respectively measure (1) whether a correct program is in the top k whenever any program scores above τ and (2) how often a correct program scoring above τ is in the top k :
[Equation]
We sweep possible values for τ to compute a precision-recall curve.
Generically, there is no ‘true’ best trade-off between these desiderata.
Pass rate.
The pass@k metric (Austin et al., 2021; Chen et al., 2021) measures the probability of k samples from P LM ( ·| prompt ) passing the ground-truth test cases, G :
[Equation]
Note that pass@k is proportional to ActualPositives (Eq. 6):
The (fraction of) problems where there is at least one correct answer in the sampled programs.
It is also conventional to combine pass@k with a scoring function that reranks the sampled programs.
This generalizes pass@k to pass@k,n , which measures the probability that, after generating n candidate programs, a correct one is in the top- k under our scoring function:
[Equation]
p ⊢ G
[Equation]
Ranking and clustering.
When the programs are grouped into clusters, we can also define pass@k,n by ranking the clusters and predicting a program from each of the top k clusters.
This works well when all programs in each cluster have the exact same score, and when clusters tend to be either 100% correct or 0% correct.
We report pass@k,n , when reranking clusters, but also analyze scoring/reranking individual programs in Appendix A.8.
Results
We study our approach on two popular datasets while using Codex models (Chen et al., 2021), seeking to answer the following research questions:
• How does our learned reranking impact raw rate of success ( pass@k,n )?
• How trustworthy and safe can we make the system (precision), and how much does that require sacrificing coverage (recall)?
• How does our learned scoring function generalize across datasets?
• How can we use the synthesized specifications to certify program correctness?
We evaluate on programming problems from the Mostly Basic Python Problems (MBPP:Austin et al.
(2021), sanitized version) and HumanEval datasets (Chen et al., 2021).
Each of these datasets contains natural language descriptions of programming problems, and holdout tests to judge program correctness.
An important difference between them is that HumanEval sometimes includes example input-outputs as part of the natural language description, while MBPP does not.
Having I/O examples in the problem description makes spec generation easier: some specs are given for free.
On the other hand, humans sometimes spontaneously mix natural language and examples (Acquaviva et al., 2021).
Therefore, using both MBPP and HumanEval gives a more robust evaluation, but we note this qualitative difference between them.
Appendix Sec.
A.1 gives further experimental setup details, such as hyperparameters and example prompts.
4. Raw accuracy improvement from reranking
To understand how well speculyzer learns to predict the best program—independent of predicting when it doesn’t know the answer—we measure pass@k,n (Fig. 3).
We assess our system using cross validation, and consider ablations (1) using only input-output testcases; and (2) using only logical relations.
We use as baselines (1) AlphaCode’s ranking function, which ranks based on cluster size (“cluster”); (2) CodeT’s ranking function, which ranks based on (testcase pass rate) √ cluster size; (3) a random baseline, which ranks all programs equally; and (4) an oracle, which always chooses a correct programs, if it exists.
We also quote numbers from Inala et al.
(2022) (which ranks programs using a separate neural net) and Chen et al.
(2022) (CodeT).
We see that both varieties of specification are valuable, but that input-outputs work better on their own than logical relations on their own.
Overall, speculyzer achieves 76.5% pass@1 on HumanEval and 73.7% pass@1 on MBPP.
To the best of our knowledge these are the highest pass@1 rates reported so far on these datasets.
On HumanEval we improve over CodeT by 11% absolute (65.8% → 76.5%), and on MBPP by 5% (67.7% → 73.7%)
(Chen et al., 2022); we also obtain a larger improvement over Inala et al.
(Different nucleus sampling parameters (Holtzman et al., 2019) cause Inala et al.
(2022) to underperform the random baseline.)
[Figure]
Calculated using n = 100 samples.
Inala et al.
(2022) does not report k = 10.
See Fig.for analogous results on other language models and sampling hyperparameters.
4. Trading off between trustworthiness and coverage
Trustworthy systems should avoid predicting any programs at all when they cannot solve a problem.
Doing so increases precision, the fraction of outputs which are correct, but at the expense of recall, the fraction of solvable problems where we output a correct solution.
Fig.
4 illustrates how one can adjust this trade-off.
For example, we can achieve 100% precision on HumanEval ( zero error rate), in exchange for dropping our recall from 82% to 33%.
Note this zero error rate does not come from our learned score function memorizing the data: we use cross validation to test each program using weights trained on other folds.
Less extreme tradeoffs are possible, such as 90% precision in exchange for 51% recall.
Our method quantitatively outperforms the alternatives on precision/recall statistics (Fig. 4, bottom), but as with pass@k , our advantage is largest on HumanEval.
We hypothesize that this is because HumanEval is a much cleaner dataset compared to MBPP, and not because HumanEval often includes input-outputs in the prompt.
Our CodeT replication controls for that difference across datasets, because it too sees input-outputs in HumanEval prompts, yet it underperforms speculyzer by a wider margin on HumanEval compared to
MBPP.
Although the best balance between precision and recall depends on the downstream application, we hope that our study invites further progress on systems that optimize this trade off, in addition to optimizing the popular pass@k .
Precision 0.
R e c a ll
HumanEval 0.
Precision 0.
R e c a ll
MBPP oracle ours
CodeTreplication Cluster random
AUC max F
R@ R@
P=.9 P= ours 0.
0.51 0.33 CodeT 0.
Cluster 0.
random 0.
AUC max F
R@ R@
P=.
P= ours 0.
0.48 0.042 CodeT 0.
Cluster 0.
random 0.
4. Generalization across datasets
Unlike recent heuristics for reranking solutions proposed by a large language model, our scheme involves learning real-valued parameters ( θ in Eq. 2).
To understand how learned parameters generalize across datasets, we compute the pass@1 rate and precision-recall stats for models trained on MBPP, but tested on HumanEval (and vice versa).
These statistics are essentially unchanged by training on different datasets (Fig. 5), varying by 4% absolute and 6% relative, indicating generalization across similar, but not identical, data distributions. test
HumanEval MBPP train HumanEval MBPP both
HumanEval
MBPP both pass@ 0.
AUC 0.
max F 0.
4. Certifying (In)correctness
No natural language program synthesizer will always produce correct programs: Therefore, the system needs to communicate what a synthesized program p computes, so that the user can confidently accept or discard it.
speculyzer does this by outputting a specification that certifies p ’s (in)correctness while being maximally informative as to p ’s behavior.
Whenever speculyzer ranks p ∗ ∈ P as the best solution to a problem, it selects a spec s ∗ ∈ S to certify the behavior of p ∗ .
The certificate s ∗ must be a true fact about p ∗ , so p ∗ ⊢ s ∗ , but should also constrain the behavior of p ∗ .
For example, the specification ∀ x : p ∗ ( x ) =
p ∗ ( x ) is vacuously true for any p ∗ , and so makes a poor certificate.
We formalize this as a rational-communication model of program synthesis (Pu et al., 2020), which means first defining a joint probability distribution over programs and specifications:
P [ p, s ] ∝ 1 [ p ⊢ s ] 1 [ p ∈ P ] 1 [ s ∈ S ].
Then, we score each specification s by the conditional probability of p ∗ given s , i.e. P [ p ∗ | s ].
Applying Bayes’ Rule and simplifying, we find that this is equivalent to ranking specs by how few other programs satisfy them, i.e. their selectivity:
[Equation]
Fig.
6 illustrates a representative programming problem and its top specification compared with a random specification.
Appendix A.11 illustrates 20 further examples.
[Figure]
[Table]
[Figure]
Contributions and Outlook
We have contributed a program synthesizer that learns to predict when it cannot solve a problem and learns to construct its own specifications that communicate what each program does.
We intend for these elements of speculyzer to increase the trust and safety of neural program synthesis and to serve as a modest step toward program synthesizers that could better collaborate with software engineers.
For this idea of better human-machine collaboration, writing correct code and documenting its correctness are paramount.
In the process, we have also improved the state-of-the-art pass@k accuracy for the HumanEval dataset, and we have laid out basic trust and safety statistics, namely recall at high precision, which we hope springboards further investigation in language model safety for source code.
Our work has important limitations.
Because speculyzer wraps around a large language model, it inherits some of their limitations, such as expensive sampling times.
Since we also run the executed code, we incur additional cost and impose security risks if that execution is not appropriately sandboxed.
Fundamentally, an approach like ours can never truly provide the same level of trust as a classic program synthesizer working from human-crafted formal-logic specifications; however, formal logic is less accessible than natural language.
Many directions remain open.
Conceptually, the idea of formal specifications as a liaison between programs and informal natural language opens up the possibility of using richer kinds of specs and verifiers.
This would allow tapping many years of effort from the programming languages community (D’silva et al., 2008; Baldoni et al., 2018), at least if we can interface such formalisms with large language models.
Using a sophisticated verifier instead of executing candidate programs could also address the security concerns and performance hit from our additional code executions.
Another direction is to combine our ideas with recent advances in HCI for program synthesis, such as Peleg et al.
(2020), which develops powerful human interaction paradigms for program synthesis.
A Appendix A. Experimental setup
Sampling from language models.
We used Codex models to draw samples using a maxtoken size of 580 for our generation of programs and specifications for both HumanEval and MBPP.
We used " \ ndef" , " \ n#" , " \ nclass" , " \ nif" , " \ nassert" , and " \ nprint" as stop tokens for our generation of programs and input-output test cases, and we used " \ n# Problem" as the stop token for our generation of the logical relations test cases.
We used zero-shot prompting for program and input-output test case generation, and few-shot prompting for the logical relations specifications generation.
In the main text we report results on the flagship Davinci Codex model, but we also obtain qualitatively similar results with the smaller Cushman Codex model (Fig. 7-8).
We drew samples from these models using nucleus sampling with temperature = 0 .
8, TopP = 0 .
95, and also tried TopP = 1 on the Davinci model.
The main text gives pass@k for Davinci w/ TopP = 0 . 95 to give a fair comparison with CodeT, which used the same sampling parameters. In practice, these hyperparameters would be tuned to maximize the number of problems where there is at least one correct sampled program. For Davinci Codex, this is TopP = 0 .
95 for MBPP and TopP = 1 for HumanEval, which we use to produce the precision-recall curves in the main text.
Appendix Fig.
7-8 illustrate pass@k and precisionrecall results for every model and parameter setting considered in this work.
Logistic regressor.
We used the Adam optimizer with 10 − 3 as the learning rate and 10 −as the weight decay.
We used 10-fold cross-validation for in-domain testing (training on
HumanEval and evaluating on HumanEval, training on MBPP and evaluating on MBPP), and we trained for 1500 epochs.
We trained for 2000 epochs for domain generalization testing (training on HumanEval and evaluating on MBPP, training on HumanEval and evaluating on MBPP).
For training, we standardize all input features to have a mean ofand a standard deviation of 1.
Verification.
We verify input-output specifications to see if they hold for each program by executing the program on the input and comparing it with the output.
Logical relations specifications require sample input(s) on which to test the relation; our prompt for logical relations causes the language model to construct such inputs, but in general one could use a fuzzer or verification tool, which we also explore (see next paragraph).
This causes verification of logical relation specifications to also reduce to program execution.
We executed generated programs and test case in a Oracle Virtual Machine as a sandbox.
We also consider checking logical relations using the property-based tester Hypothesis, which systematically probes the input space of each logical relation based on analyzing the underlying source code.
This allows us check whether a logical relation might hold on all inputs instead of just those considered by Codex. Depending on experimental condition, this is true for about 40%-70% of the logical relations (Tbl. 1). When a logical relation is true over the whole space of inputs, we can return a stronger spec to the user which asserts the synthesized predicate holds for all inputs, instead of simply showing the inputs on which the relation was verified. [Table]
A. Supplemental Discussion: Soundness & Completeness
The formal methods community has traditionally sought methods that are sound and complete Baier & Katoen (2008).
Soundness means that whenever a program satisfies a spec, we identify it as such.
Completeness means that every spec we identify as valid for a program is indeed entailed by that program.
Generally, speculyzer ’s verification step to overapproximates the set of specs a program satisfies, achieving completeness at the expense of soundness.
This overapproximation only occurs for logical relations: we’re sound and complete for input-outputs, because they assert a property of the program only on a single input.
But because the language model constructs specific inputs on which to run the logical relations, we cannot assert that they hold generically for every single possible input, unless we perform more sophisticated verification steps such as the property-based methods described in the preceding section.
The practical impact of this unsoundness for logical relations is that a human user has to inspect the candidate inputs on which the language model probes the relation.
In practice, we find that the neural network generates fairly representative inputs (Appendix A.11), something that AlphaCode
(Li et al., 2022) also found.
In principle, nothing precludes running a model checker or solver to check that the logical relations hold over every single possible input, which would make the method sound and complete w.r.t.
the programs and specifications.
However, this would not eliminate the need for a human to examine the AI-generated specifications: ultimately, the true spec is in natural language, and there is no sound and complete verifier against informal language.
A. Dataset Statistics
Below we show representative dataset statistics for Davinci Codex with temperature 0.8 and topP=1.
[Table]
average # of test cases per program
[Table]
A. Example Zero-Shot Prompts for program generation
For MBPP, to generate programs, we converted the natural language prompt to a function by adding in the prompt as a docstring for a function with the name of the function called in the ground-truth test cases.
We used the HumanEval prompts as is.
Two examples of zero-shot prompts used for program generation are as follows:
A.4. HumanEval
First example: def is_happy (s) """Youaregiven a string s. Your task is to check if the string is happy or not.
A string is happy if its length is at least 3 and every 3 consecutive letters are distinct ,
→ For example: is_happy(a) =
> False is_happy(aa) = > False k= k= 0. p a ss @
k
HumanEval k= k= 0. p a ss @ k MBPP oracle ours ours, i/
o only ours, rels only cluster
CodeT replication random
Codex Davinci, temp=0.8,topP=0.
k= k= 0.
p a ss @
k
HumanEval k= k= 0.
p a ss @
k
MBPP oracle ours ours, i/
o only ours, rels only cluster
CodeT replication random
Codex Davinci, temp=0.8,topP=1. k= k= 0.
p a ss @ k HumanEval k= k= 0. p a ss @
k
MBPP oracle ours ours, i/
o only ours, rels only cluster
CodeT replication random
Codex Cushman, temp=0.8,topP=0.
Precision 0.
R e c a ll
HumanEval 0.
Precision 0.
R e c a ll
MBPP oracle ours ours, rels only ours, i/o only
Cluster
CodeTreplication random
Codex Davinci, temp=0.8,topP=0.
Precision
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
HumanEval 0.
Precision 0.
R e c a ll
MBPP oracle ours ours, rels only ours, i/o only
Cluster
CodeTreplication random
Codex Davinci, temp=0.8,topP=1.
[Figure]
[Figure]
Precision
[Figure]
[Figure]
HumanEval 0.
Precision
[Figure]
[Figure]
[Figure]
[Figure]
MBPP oracle ours ours, rels only ours, i/o only
Cluster
CodeTreplication random
Codex Cushman, temp=0.8,topP=0.
is_happy(abcd) => True is_happy(aabb)
=
> False is_happy(adb) =
> True is_happy(xyy) =
> False """
Second example: def fix_spaces(text): """
Givenastringtext, replace all spaces in it with underscores, and if a string has more than 2 consecutive spaces, then replace all consecutive spaces withfix_spaces("Example")
==
"Example" fix_spaces("Example 1") == "Example_1" fix_spaces(" Example 2")
== "_Example_2" fix_spaces(" Example 3") =
= "_Example-3" """
A.4.
MBPP
First example: def sum_range_list(list1 : list, m : int, n : int) -> int: """
Write a function to find the sum of numbers in a list within a range specified by two indices.
, → """
Second example: def diff_even_odd(list1 : list) -> int: """
Write a function to find the difference of the first even and first odd number of a given list.
, → """
A. Example Zero-Shot Prompts for input-output generation
We extracted input-output test cases by generating n = 100 times per HumanEval/MBPP prompt, then extracting each distinct single-line test case from each generation.
We do this because each generation may produce multiple test cases, and we aimed to test each program on a single test case.
For our test case prompts, we used the prompts to generate programs from MBPP and HumanEval, and we added in a pass # To-do: implement statement, a line with a comment asking Codex to # Check if func name works and another line to asking Codex to assert func name( .
A.5. HumanEval First
example: def is_happy(s): """Youaregiven a string s.
Your task is to check if the string is happy or not.
A string is happy if its length is at least 3 and every 3 consecutive letters are distinct ,
→ For example: is_happy (a) =>
False is_happy (aa) =>
False is_happy(abcd) =>
True is_happy(aabb) =
> False is_happy(adb) =
> True is_happy(xyy) =
> False """ pass # To-do: implement # Check if is_happy works assert is_happy(
Second example: def fix_spaces(text): """
Givenastringtext, replace all spaces in it with underscores, and if a string has more than 2 consecutive spaces, then replace all consecutive spaces withfix_spaces("Example")
==
"Example" fix_spaces("Example 1") == "Example_1" fix_spaces(" Example 2")
== "_Example_2" fix_spaces(" Example 3") =
= "_Example-3" """ pass # To-do: implement # Check if fix_spaces works assert fix_spaces(
A.5.
MBPP
First example: def sum_range_list(list1 : list, m : int, n : int) -> int: """
Write a function to find the sum of numbers in a list within a range specified by two indices.
, → """ pass # To-do: implement # Check if sum_range_list works assert sum_range_list(
Second example: def diff_even_odd(list1 : list) -> int: """
Write a function to find the difference of the first even and first odd number of a given list.
, → """ pass # To-do: implement # Check if diff_even_odd works assert diff_even_odd(
A. Few-Shot Prompt for logical relations spec generation
We use two-shot examples prompting to guide the model to tests various kinds of properties.
A HumanEval #
Problemfrom typing import List def filtered_even_integers(input_list: List([int]) -> List[int]: """ Given a list of integers, return a list that filters out the even integers.
, → >>> filtered_even_integers([1, 2, 3, 4]) [1, 3] >>> filtered_even_integers([5, 4, 3, 2, 1]) [5, 3, 1] >>> filtered_even_integers([10, 18, 20])
[] """ #TODO pass # Testdef test_filtered_even_integers(input_list: List()): """ Given an input `input_list`, test whether the function `filtered_even_integers` is implemented correctly.
, → """ output_list = filtered_even_integers(input_list) # check if the output list only contains odd integers for integer in output_list: assert integer %
2 ==
# check if all the integers in the output list can be found in the input list , → for integer in output_list: assert integer in input_list # run the testing function `test_filtered_even_integers` on a new testcase test_filtered_even_integers([31, 24, 18, 99, 1000, 523, 901])
# Problemdef repeat_vowel(input_str: str) -> str: """ Return a string where the vowels (`a`, `e`, `i`, `o`, `u`, and their capital letters) are repeated twice in place. , → >>> repeat_vowel('abcdefg') 'aabcdeefg' >>>repeat_vowel('Amy
Emily Uber') '
AAmy EEmiily UUbeer' """ #TODO pass # Testdef test_repeat_vowel(input_str: str) : """ Given an input `input_str`, test whether the function `repeat_vowel` is implemented correctly.
, → """ output_str = repeat_vowel(input_str) vowels =
['a', 'A', 'e', 'E', 'i', 'I', 'o', 'O', 'u', 'U'] # check if the number of vowels in the output string is doubled # First get the number of vowels in the input number_of_vowels_input = sum([input_str.count(vowel) for vowel in vowels]) , → # Then get the number of vowels in the output number_of_vowels_output = sum([output_str.count(vowel) for vowel in vowels]) , → assert number_of_vowels_input * 2 =
= number_of_vowels_output
# run the testing function `test_repeat_vowel` on a new testcase test_repeat_vowe('ABCDEabcdeABCDE YOUUOY')
A.6.
MBPP # Problemfrom typing import List # Given a list of integers, return a list that filters out the even integers.
, → def filtered_even_integers(input_list: List([int]) -> List[int]: pass # To-do: Implement # Testdef test_filtered_even_integers(input_list: List()): """ Given an input `input_list`, test whether the function `filtered_even_integers` is implemented correctly.
, → """ output_list = filtered_even_integers(input_list) # check if the output list only contains odd integers for integer in output_list: assert integer %
2 ==
# check if all the integers in the output list can be found in the input list , → for integer in output_list: assert integer in input_list # run the testing function `test_filtered_even_integers` on a new testcase test_filtered_even_integers([31, 24, 18, 99, 1000, 523, 901])
# Problem# Return a string where the vowels (`a`, `e`, `i`, `o`, `u`, and their capital letters) are repeated twice in place , → def repeat_vowel(input_str: str) -> str: pass # To-do: Implement # Testdef test_repeat_vowel(input_str: str) : """ Given an input `input_str`, test whether the function `repeat_vowel` is implemented correctly.
, → """ output_str = repeat_vowel(input_str) vowels =
['a', 'A', 'e', 'E', 'i', 'I', 'o', 'O', 'u', 'U'] # check if the number of vowels in the output string is doubled # First get the number of vowels in the input number_of_vowels_input = sum([input_str.count(vowel) for vowel in vowels]) , → # Then get the number of vowels in the output number_of_vowels_output = sum([output_str.count(vowel) for vowel in vowels]) , → assert number_of_vowels_input * 2 =
= number_of_vowels_output
# run the testing function `test_repeat_vowel` on a new testcase test_repeat_vowe('ABCDEabcdeABCDE YOUUOY')
A. Multilayer Perceptron Model
In the main text we describe a linear model that computes confidence scores for each program based on features of the verification results.
We also tried a multilayer perceptron with a single hidden layer and five hidden units: a small model, because we have a low dimensional problem with at most a few hundred training problems.
As shown in Fig.
9, the multilayer perceptron does not actually do better on held out data, as measured by pass@k.
A. Ranking individual programs vs ranking clusters
In the main text we describe pass@k results based on first clustering the programs according to which specifications they pass, and drawing k programs from the top k clusters.
A simpler approach is to simply return the top k ranked programs.
Overall this gives inferior pass@k for k > 1, and by design has no effect when k = 1 (Fig. 10).
A. Threshold Calibration
Our scoring function comes from logistic regression, which is a probabilistic discriminative model.
Maximum likelihood training encourages it to be well-calibrated.
Calibration means that the classifier not only discriminates positive/negative examples correctly but also, whenever it predicts probability x of a positive label, approximately x of the examples are actually positive examples.
In our setting, this means when the model predicts 90% confidence that a program is correct, then about 90% of the time the program actually is correct.
We experimentally confirmed this calibration property, which allows tuning the threshold τ to achieve the desired precision.
The free parameter τ acts as a threshold on the confidence score needed to output a program.
Because our scoring function comes from logistic regression, the threshold τ also acts as a threshold on how high the predicted probability that a program is correct has to be before we consider it to be a candidate solution.
In particular, the logistic regressor predicts the probability of program correctness as σ (score), so thresholding score by τ corresponds to thresholding probability by σ − 1 ( τ ).
Thus, if our classifier is well-calibrated, we can set the threshold τ to the desired precision.
Indeed, our model is out-of-the-box well-calibrated, as illustrated in Fig.
[Figure]
a c t u a l p r o b a b ili t y
[Figure]
A. Transformation of input problems to logical relations prompts
Here we show how to transform the input problem to the prompt used for generating logical relations.
MBPP transformation First we parse the input problems from MBPP dataset and get the string representation of library imports, function name, function parameters, return type, and English problem description.
We denote them as imports , func name , parameter format , return type , and description respectively.
Then we use the template shown in Figure 13 and Figure 12 for input-output and logical relations respectively.
The parsed string from the input problem would then be inserted to the placeholder accordingly.
# Problem{imports} def{func_name}({", ".join(parameter_format)}) -> {return_type}: """ {description} """ pass#To-do: implement # Testdef test_{func_name}(
{imports} def {func_name}({", ".join(parameter_format)}) -> {return_type}: """ {description} """ pass#To-do: implement # Check if {func_name} works assert {func_name}('
Then, for the logical relations, we prepend the resulting string with the few shot example string shown in A.6.2.
For the input-output, we strip out the blank lines prefix if import is empty.
HumanEval Transformation Similar to the above MBPP transformation, we parse the input problems from HumanEval dataset and get the string representation of function definition plus English description and function name.
We denote them as function definition with description and func name .
Then we insert these into the template shown in Figure 14 and Figure 15 for input-output and logical relations respectively.
{function_definition_with_description} pass # Check if {func_name} works assert {func_name}(
# Problem{function_definition_with_description} pass#To-do:implement # Testdef test_{func_name}(
Finally, we prepend the resulting string with the few shot example string shown in A.6.1.
A.
Examples of Top Certificates versus Random Certificates
Below we show example HumanEval problems, our top ranked sample, our top ranked specification certifying the (in)correctness of that sample, and a random specification.
There are 20 such examples, randomly chosen; half show input-output specifications, and the other half show logical relations.
Example synthesized program 1 (sample is correct) from typing import List def mean_absolute_deviation(numbers: List[float]) -> float: """ For a given list of input numbers , calculate Mean Absolute
Deviationaroundthe mean of this dataset.
Mean Absolute Deviation is the average absolute difference between eachelement and a centerpoint (mean in this case):
MAD =
[Equation]
>>
[Equation]
mean_absolute_deviation ([1.0, 2.0, 3.0, 4.0]) 1.
[Equation]
" mean = sum(numbers) / len(numbers) mad =for number in numbers: mad += abs(number - mean) return mad / len(numbers)
def test_mean_absolute_deviation(numbers: List[float]): """ Given an input list `numbers `, test whether the function ` mean_absolute_deviation ` is implemented correctly. "
"" output = mean_absolute_deviation(numbers) # check if the result of mean_absolute_deviation is the same as numpy.mean(np.abs(numbers - np.mean(numbers))) try: import numpy as np # use this boolean variable to determine whether the test is passedtest_pass =
output =
= np.mean(np.abs(numbers - np.mean(numbers )))
except ImportError: print('make sure you have numpy installed')
# if numpy is not installed , the test will not be run test_pass =
False assert test_pass # run the testing function `test_mean_absolute_deviation ` on a new testcase test_mean_absolute_deviation ([48.35 , 48.45, 48.85, 49.15, 49.35, 49.55, 50.15, 50.55, 51.55, 54.45])
Random relational spec for program
def test_mean_absolute_deviation(numbers: List[float]) -> float: """ Given an input `numbers `, test whether the function ` mean_absolute_deviation ` is implemented correctly. "
"" #TODOpass # run the testing function `test_mean_absolute_deviation ` on a new testcase test_mean_absolute_deviation ([1.0, 2.0, 3.0, 4.0])
Example synthesized program 2 (sample is correct) from typing import List def filter_by_substring(strings: List[str], substring: str) -> List[ str]: """Filter an input list of strings only for ones that contain given substring >>> filter_by_substring ([], 'a')
[] >>> filter_by_substring(['abc', 'bacd ', 'cde', 'array '], 'a'
[Table]
[Table]
def test_filter_by_substring(input_list: List[str], input_substring: str): """Given an input `input_list ` and `input_substring `, test whether the function `filter_by_substring ` is implemented correctly."
"" output_list = filter_by_substring(input_list , input_substring) # check if all the elements in the output list do contain the given substring for string in output_list: assert input_substring in string # run the testing function `test_filter_by_substring ` on a new testcase test_filter_by_substring (['ABCDE', 'BACDE', 'CDE', 'ARRAY'], 'BC')
Random relational spec for program
def test_filter_by_substring(strings: List[str], substring: str) -> List[str]: """Given a list of `strings ` and an input `substring `, test whether the function `filter_by_substring ` is implemented correctly."
"" output_strings = filter_by_substring(strings , substring) # check if every element returned in the output list contains the given substring for element in output_strings: assert substring in element # run the testing function `test_filter_by_substring ` on a new testcase test_filter_by_substring (['abc', 'bcd', 'cde', 'adb', 'aed'], 'ae')
Example synthesized program 3 (sample is incorrect) from typing import List ,
Tuple def rolling_max(numbers: List[int]) -> List[int]: """ From a given list of integers , generate a list of rolling maximum element found until given moment in the sequence.
>>> rolling_max([1, 2, 3, 2, 3, 4, 2]) [1, 2, 3, 3, 3, 4, 4] """result: List[int]
= [] current_max: int = numbers [0] for i in numbers: if i > current_max: current_max = i result.append(current_max) return result
[Table] [Table]
def
[Table]
list
[Table]
List
[Table]
int])
[Table]
for i in
[Table]
(len(
[Table] [Table]
: assert
[Table]
[Table] [Table]
[Table] [Table]
# run the
[Table]
function `
[Table]
_rolling_max
[Table]
a new
[Table]
max([
[Table] [Table]
import random def test_rolling_max(numbers: List[int]): """ Given an input `numbers `, test whether the function ` rolling_max ` is implemented correctly. "
"" rolling_max_list = rolling_max(numbers) # check if the length of the output list is correct assert len(rolling_max_list) =
= len(numbers)
# check if the output list is correct for i in range(len(rolling_max_list)): assert rolling_max_list[i] =
= max(numbers[:i + 1])
# run the testing function `test_rolling_max ` on a new testcase test_rolling_max(numbers =
[random.randint(0, 1000) for i in range (100)])
Example synthesized program 4 (sample is correct) def count_upper(
[Table]
Givena
[Table]
s, count the number of uppercase
[Table]
in even indices.
For example: count_upper('aBCdEf
[Table] [Table]
[Table]
Example synthesized program 5 (sample is incorrect) def get_max_triples(n):"""
Youaregivenapositive integer n. You have to create an integer array a of length n.
For each i (1 <= i <= n), the value of a[i] =
i * i - i + 1.
Return the number of triples (a[i], a[j], a[k]) of a where i < j < k, and a[i] + a[j] + a[k] is a multiple of 3.
Example : Input: n =
Output: 1 Explanation: a=[1,3,7, 13, 21] The only valid triple is (1, 7, 13).
"""ifn < 3: return 0 cache = {} for i in range(3, n+1): residue = i*i - i + 1 %if residue == 0: cache[residue] =else:if residue in cache: cache[residue] += 1 else: cache[residue] = 1 res =for i in cache: if cache[i] >= 2: res += get_combinations(cache[i], 2) return res def test_get_max_triples(n):"""Givenaninput`n`,test whether the function `get_max_triples ` is implemented correctly. "
"" output = get_max_triples(n)
# check if the output is of the correct type assert isinstance(output , int)
a =
[i * i - i + 1 for i in range(1, n+1)]
# check if the output is of the correct value number_of_valid_triples =for i in range(1, n): for j in range(i+1, n): if a[i] + a[j] % 3 == 0: number_of_valid_triples +=assert number_of_valid_triples =
= output # run the testing function `test_get_max_triples ` on a new testcase test_get_
[Table]
Random relational spec for programdef test_get_max_triples(n: int): """ Given an input `input_str `, test whether the function ` get_max_triples ` is implemented correctly. "
"" #TODOpass test_get_max_triples (34) # should print 92 test_get_max_triples (24) # should printtest_get_max_triples (14) # should print 18 test_get_max_triples (23) # should printtest_get_max_triples (35) # should print 96 test_get_max_triples (43) # should printtest_get_max_triples (2) # should print 0 test_get_max_triples (32) # should printtest_get_max_triples (15) # should print 18 test_get_max_triples (21) # should printtest_get_max_triples (36) # should print
Example synthesized program 6 (sample is incorrect) from typing import List def separate_paren_groups(paren_string: str) -> List[str]: """ Input to this function is a string containing multiple groups of nested parentheses.
Your goal is to separate those group into separate strings and return the list of those.
Separate groups are balanced (each open brace is properly closed) and not nested within each other Ignore any spaces in the input string.
>>> separate_paren_groups('( ) (( ))
['()', '(())', '(()())'] """return []
[Table]
check if the output is a list assert isinstance(output_list , list) for individual_group in output_list: # check if each individual group has the same number of open and close brackets assert individual_group.count('(') =
= individual_group.count(' )')
# check if separate groups are not nested within each other assert individual_group.count('()')
=
=
# run the testing function `test_separate_paren_groups ` on a new testcase test_separate_paren_groups('( ) (( ))
Example synthesized program 7 (sample is correct) def compare_one(a, b): """
Create a function that takes integers , floats , or strings representingrealnumbers , and returns the larger variable in its given variable type.
Return None if the values are equal.
Note: If a real number is represented as a string , the floating point might be .
or , compare_one(1, 2.5) -> 2.5 compare_one(1, "2,3") -> "2,3" compare_one ("5,1", "6") -> "6" compare_one ("1", 1) -> None """#create a list of all variables without types list_of_vars =
[a, b] list_of_vars =
[str(i) for i in list_of_vars] # remove commas for i, val in enumerate(list_of_vars): if ',
' in val: list_of_vars[i] = val.replace(",", ".")
# compare variables and return if float(list_of_vars [0]) <
float(list_of_vars [1]): return b elif float(list_of_vars [0]) >
float(list_of_vars [1]): return a else: return None
Top ranked relational spec for programdef test_compare_one (): """
Test whether the function `compare_one ` is implemented correctly. """ assert compare_one(1, 2.5) =
= 2.5 assert compare_one(1, "2,3") =
= "2,3" assert compare_one("5,1", "6")
== "6" assert compare_one("1", 1) =
=
None test_compare_one ()
Random relational spec for programdef test_compare_one(a, b): """ Given an inputs `a` and `b`, test whether the function ` compare_one ` is implemented correctly. "
"" output = compare_one(a, b) if a > b: assert output ==
a elif a < b: assert output ==
b elif a ==
b: assert output =
=
None # run the testing function `test_compare_one ` on a new testcase test_compare_one(1, 2.5)
Example synthesized program 8 (sample is correct) def is_prime(n):"""Returntrue if a given number is prime , and false otherwise.
>>> is_prime (6) False >>>is_prime (101)
True >>> is_prime (11)
True >>> is_prime (13441)
True >>> is_prime (61) True >>> is_prime (4) False >>>is_prime (1) False """ if n == 2: return True if n < 2 or (n % 2) =
= 0: return False return not any(n % i =
=
for i in range(3, int(n ** 0.5) + 1, 2))
[Table]
Example synthesized program 9 (sample is incorrect) def closest_integer(value):'''
Createafunctionthattakes a value (string) representing a numberandreturns the closest integer to it.
If the number is equidistantfromtwointegers , round it away from zero.
Examples>>>closest_integer ("10") 10>>> closest_integer ("15.3")
Note:Rounding away from zero means that if the given number is equidistantfromtwointegers , the one you should return is the one that is thefarthest from zero.
For example closest_integer ("14.5") should return 15 and closest_integer (" -14.5") should return -15. '''
return round(float(value))
Top ranked relational spec for program
def test_closest_integer(value):'''
Givenavalue,testwhetherthe function `closest_integer ` is implemented correctly.
'''output_val = closest_integer(value) assert type(output_val) is int and round(float(value))
=
= output_val # run the testing function `test_closest_integer ` on a new testcase test_closest_integer("14.5")
test_closest_integer("-14.5")test_closest_integer("14.4")
test_closest_integer("-14.4")
Random relational spec for programdef test_closest_integer(value):'''
Testwhetherthefunction`closest_integer ` is implemented correctly.
#TODO pass test_closest_integer("10.1")
Example synthesized program 10 (sample is correct) def can_arrange(arr):"""Createafunction which returns the largest index of an element whichisnot greater than or equal to the element immediately preceding it.
If no such element exists then return -1.
The given array will not containduplicate values.
Examples:can_arrange ([1,2,4,3,5]) =
can_arrange ([1,2,3]) = -1 """ for i in range(len(arr)-1, 0, -1): if arr[i-1] > arr[i]: return i return -
[Table]
def test_can_arrange(arr):"""Givenaninput`arr`, test whether the function `can_arrange ` is implemented correctly. "
"" output = can_arrange(arr) # Check if the function returns -1 when there are no elements that are not greater than or # equal to the element immediately preceding it.
if (all(arr[i] >= arr[i-1] for i in range(1, len(arr)))): assert output == - # Check if the function returns the largest index of an element which is not greater # than or equal to the element immediately preceding it.
else: assert output ==
max([i for i in range(1, len(arr))
if arr[i] < arr[i-1]]) # run the testing function `test_can_arrange ` on a new testcase test_can_arrange ([1, 2, 4, 3, 5]) test_can_arrange ([1, 2, 3]) test_can_arrange ([5, 2, 3]) test_can_arrange ([5, 2, 2, 2, 2]) test_can_arrange ([9, 8, 7, 6, 5, 4, 3, 2, 1])
Random relational spec for programdef test_can_arrange(arr):"""Givenanarray`arr`, test whether the function `can_arrange ` isimplemented correctly.
"""output = can_arrange(arr) if output == -1: assert len(arr) =
= 0 or all(arr[i + 1] <= arr[i] for i in range(len(arr) - 1)) else: assert 0 <= output < len(arr) assert output == 0 or arr[output] < arr[output - 1] assert output
== len(arr) - 1 or arr[output] >= arr[outpus + 1] # run the testing function `test_can_arrange ` on a new testcase test_can_arrange ([7, 6, 5, 4, 3, 2, 1])
Example synthesized program 11 (sample is correct) def specialFilter(nums):"""Writeafunctionthat takes an array of numbers as input and returnsthenumber of elements in the array that are greater than 10 and bothfirst and last digits of a number are odd (1, 3, 5, 7, 9).
For example: specialFilter ([15, -73, 14, -15]) =>
specialFilter ([33, -2, -3, 45, 21, 109])
=> 2 """ return len([num for num in nums if (num > 10 and (num % 10 in {1, 3, 5, 7, 9} and num // 10 ** (len(str(num)) - 1) in {1, 3, 5, 7, 9}))]) return len(list(filter(lambda x: bool(x > 10 and (x % 10 in {1, 3, 5, 7, 9} and x // 10 ** (len(str(x)) - 1) in {1, 3, 5, 7, 9})), nums)))
Top ranked input-output spec for programassert specialFilter ([33, 23, 75, 21, -1, 109]) =
=
Random input-output spec for programassert specialFilter([1, 2, 3, 4])
==
[Table]
def make_a_pile(n):"""
Givenapositive integer n, you have to make a pile of n levels of stones.
[Table] [Table]
[Table]
[Table]
is:
number if n is odd.
- the next even number if n is even.
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Examples:>>>make_a_pile (3) [3, 5, 7] """ stones =
[] for level in range(1, n + 1): stones.append(n + 2 * level - 2) return stones
[Table]
assert make_a_pile (9) =
=
[Table]
assert make_a_pile (4) =
=
[4, 6, 8, 10], "Wrong implementation"
Example synthesized program 13 (sample is correct) def do_algebra(operator , operand): """
Given two lists operator , and operand.
The first list has basic algebra operations , and the second list is a list of integers.
Use the two given lists to build the algebric expression and return the evaluation of this expression.
The basic algebra operations: Addition ( + )
Subtraction ( - ) Multiplication ( * )
Floor division ( // )
Exponentiation ( ** )
Example:operator['+', '*', '-'] array =
[2, 3, 4, 5] result = 2 + 3 * 4 -=> result =
Note:The length of operator list is equal to the length of operand list minus one.
Operand is a list of of non -negative integers.
Operator list has at least one operator , and operand list has at least two operands.
"""iflen(operator and len
: expression else:return None
Top ranked input-output spec for program
assert do_algebra(['+', '*'], [2, 3, 4, 5]) =
=
Random input-output spec for programassert do_algebra(['+', '*', '-'], [2, 3, 4, 5])
==
[Table] [Table]
Explanation of the strategy
[Table]
# if n == 0: return 1 because 2ˆ0 = 1 # if n == 1: return 2 because 2ˆ1 = 2 % p
=
# if n == 2: return 4 because 2ˆ2 = 4 % p = 4 # if n == 3: return 3 because 2ˆ3 = 8 % p = 3 (p is prime , so 2ˆp = 2) # if n == 4: return 1 because 2ˆ
= 16 % p =
# if n == 5: return 2 because 2ˆ
= 32 % p = 2 # if n == 6: return 4 because 2ˆ6 = 64 % p =
# if n == 7: return 3 because 2ˆ
= 128 % p = 3 (p is prime , so 2ˆ p = 2)
# if n == 8: return 1 because 2ˆ
= 256 % p = 1 # if n == 9: return 2 because 2ˆ9 = 512 % p =
# ... # if n == 2k: return 2ˆ{2k}
= 2ˆ{k}ˆ2 % p =
(2ˆk % p)ˆ2 % p # if n == 2k + 1: return 2ˆ{2k+1} = 2 * 2ˆ{2k} =
(2ˆk % p)ˆ2 % p = 2 * (2ˆk % p)ˆ2 % p # if n == 2k + 1: 2ˆn % p = 2 * 2ˆ{2k} % p = 2 * (2ˆk % p)ˆ2 % p # if n == 2k: 2ˆn % p =
(2ˆk % p)ˆ2 % p if n ==
: returnelse:if n % 2 == 0: half =
, p return (half ** 2) % p else:return (
[Table]
Example synthesized program 15 (sample is correct)
def triangle_area(a, h): """Given length of a side and high return area for a triangle.
>>> triangle_area(5, 3) 7.
"""return (a * h) /
Top ranked input output spec for programassert triangle_area (1.5, 1.5) =
= 1.
Random input-output spec for programassert triangle_area(5, 3) =
= 7.5, 'Test failed'
Example synthesized program 16 (sample is correct) def fibfib(n: int): """The FibFib number sequence is a sequence similar to the
Fibbonacci sequnece that's defined as follows: fibfib (0) =
=fibfib (1) =
= 0 fibfib (2) =
=fibfib(n) =
= fibfib(n-1)
+ fibfib(n-2) + fibfib(n-3).
Please write a function to efficiently compute the n-th element of the fibfib number sequence. >>> fibfib (1) 0>>> fibfib (5) 4>>> fibfib (8) 24""" ifn < 2: returnelif n == 2: returnelse:f0 =f1 = 0 f2 =for i in range(3, n+1): f0, f1, f2 = f1, f2, f0+f1+f return f
Top ranked input-output spec for programassert fibfib (8) == 24, "fibfib 8" ''' 6.
Write a function to return the most common character in a string.
Inputs: str: string to be evaluated Outputs: character that occurs most frequently in the string Notes: This problem can be solved in O(n) time using a hash map.
Examples: - "hello world" -> 'l' - "aabbcc" -> 'a' - " mississippi" -> 'i' '''
Random input-output spec for programassert fibfib (5) =
= 4, "Did you implement fibfib(n) correctly?"
Example synthesized program 17 (sample is correct) def fib(n: int): """ Return n-th Fibonacci number.
>>> fib(10)>>> fib(1)>>> fib(8)""""*** YOUR CODE HERE ***" a, b = 0, 1 while n > 0: a, b = b, a + b n -=return a
Top ranked input-output spec for programassert fib (1000) ==
Random input-output spec for programassert fib(10) =
= 55, fib(10)
Example synthesized program 18 (sample is incorrect) def prime_length(string): """Write a function that takes a string and returns True if the stringlength is a prime number or False otherwise
Examplesprime_length('Hello ')
=
=
True prime_length('abcdcba ') =
=
True prime_length('kittens ') =
=
True prime_length('orange ') =
= False """ length_str = len(string) for i in range(2, length_str - 1): if length_str % i =
= 0: return False return True assert prime_length('v') =
=
True , "Failed prime_length test 5" input assert prime_length('orange') =
=
False , 'Check function definition!'
Example synthesized program 19 (sample is correct) from typing import List def remove_duplicates(numbers: List[int]) -> List[int]: """ From a list of integers , remove all elements that occur more than once.
Keep order of elements left the same as in the input.
>>> remove_duplicates ([1, 2, 3, 2, 4])
[1, 3, 4] """num_count = {} for num in numbers: num_count[num] = num_count.get(num , 0)
+new_list =
[] for num in numbers: if num_count[num] =
= 1: new_list.append(num) return new_list assert remove_duplicates ([1, 2, 1, 2, 1, 2, 3])
=
=
Random input-output spec for programassert remove_duplicates ([1, 2, 1, 2])
=
=
Example synthesized program 20 (sample is incorrect) def is_bored(S):"""
You'llbegiven a string of words , and your task is to count the numberofboredoms.
A boredom is a sentence that starts with the word "I ".Sentences are delimited by '.', '?' or '!'.
For example: >>> is_bored("Hello world") 0>>> is_bored("The sky is blue.
The sun is shining.
I love this weather") """boredoms =# replace .
or !
or ?
with .
to simplify this problem S =
S.replace('.'
S = S.replace('!','! ')
S =
S.replace
('?','? ') sentences = S.split(' ') for sentence in sentences: if sentence.startswith('I'):boredoms=boredoms+ return boredoms
Top ranked input-output spec for program
assert is_bored("I have no idea what I'm doing")
==
Random input-output spec for programassert is_bored("I love this weather.")
==