rm table
Browse files- evaluation/intro.txt +5 -1
evaluation/intro.txt
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
A popular evaluation framework for code generation models is the [pass@k](https://huggingface.co/metrics/code_eval) metric on [HumanEval](https://huggingface.co/datasets/openai_humaneval) dataset, which was introduced in [Codex paper](https://arxiv.org/pdf/2107.03374v2.pdf). The dataset includes 164 handwritten programming problems. In the pass@k metric, k code samples are generated per problem, and a problem is considered solved if any sample passes the unit tests and the total fraction of problems solved is reported.
|
| 2 |
In most papers, 200 candidate program completions are sampled, and pass@1, pass@10, and pass@100 are computed using an unbiased sampling estimator. Table 1 below shows the HumanEval scores of CodeParrot, InCoder, PolyCoder, CodeGen and Codex (not open-source).
|
| 3 |
|
|
|
|
|
|
|
| 4 |
Model | pass@1 | pass@10 | pass@100|
|
| 5 |
|-------|--------|---------|---------|
|
| 6 |
|CodeParrot (110M) | 3.80% | 6.57% | 12.78% |
|
|
@@ -18,4 +20,6 @@ In most papers, 200 candidate program completions are sampled, and pass@1, pass@
|
|
| 18 |
|||||
|
| 19 |
|Codex (25M)| 3.21% | 7.1% | 12.89%|
|
| 20 |
|Codex (300M)| 13.17%| 20.37% | 36.27% |
|
| 21 |
-
|Codex (12B)| 28.81%| 46.81% | 72.31% |
|
|
|
|
|
|
|
|
|
| 1 |
A popular evaluation framework for code generation models is the [pass@k](https://huggingface.co/metrics/code_eval) metric on [HumanEval](https://huggingface.co/datasets/openai_humaneval) dataset, which was introduced in [Codex paper](https://arxiv.org/pdf/2107.03374v2.pdf). The dataset includes 164 handwritten programming problems. In the pass@k metric, k code samples are generated per problem, and a problem is considered solved if any sample passes the unit tests and the total fraction of problems solved is reported.
|
| 2 |
In most papers, 200 candidate program completions are sampled, and pass@1, pass@10, and pass@100 are computed using an unbiased sampling estimator. Table 1 below shows the HumanEval scores of CodeParrot, InCoder, PolyCoder, CodeGen and Codex (not open-source).
|
| 3 |
|
| 4 |
+
<div align="center">
|
| 5 |
+
|
| 6 |
Model | pass@1 | pass@10 | pass@100|
|
| 7 |
|-------|--------|---------|---------|
|
| 8 |
|CodeParrot (110M) | 3.80% | 6.57% | 12.78% |
|
|
|
|
| 20 |
|||||
|
| 21 |
|Codex (25M)| 3.21% | 7.1% | 12.89%|
|
| 22 |
|Codex (300M)| 13.17%| 20.37% | 36.27% |
|
| 23 |
+
|Codex (12B)| 28.81%| 46.81% | 72.31% |
|
| 24 |
+
|
| 25 |
+
</div>
|