update

evaluation/intro.txt

@@ -4,6 +4,7 @@ In most papers, 200 candidate program completions are sampled, and pass@1, pass@
 
 | Model | pass@1 | pass@10 | pass@100|
 |-------|--------|---------|---------|
+|CodeParrot (110M) | 3.80% | 6.57% | 12.78% | 
 |CodeParrot (1.5B) | 3.58% | 8.03% | 14.96% |
 |||||
 |InCoder (6.7B) | 15.2% | 27.8% | 47.00% |
@@ -15,24 +16,8 @@ In most papers, 200 candidate program completions are sampled, and pass@1, pass@
 |GPT-neo (1.5B)| 4.79% | 7.47% | 16.30% |
 |GPT-J (6B)| 11.62% | 15.74% | 27.74% |
 
-
-To better understand how pass@k metric works, we will illustrate it with some examples. We select 4 problems from the HumanEval dataset and see how the model performs and which code completions pass the unit tests. We will use CodeParrot 🦜  with the three problem below:
-
-```python
-
-from typing import List
-
-
-def has_close_elements(numbers: List[float], threshold: float) -> bool:
-    """ Check if in given list of numbers, are any two numbers closer to each other than
-    given threshold.
-    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
-    False
-    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
-    True
-    """
-
-````
+<br/>
+To better understand how pass@k metric works, we will illustrate it with some examples. We select two problems from the HumanEval dataset and see how the model performs and which code completions pass the unit tests. We will use CodeParrot 🦜 (110M)  with the two problems below:
 
 ```python
 
@@ -47,7 +32,6 @@ def separate_paren_groups(paren_string: str) -> List[str]:
     >>> separate_paren_groups('( ) (( )) (( )( ))')
     ['()', '(())', '(()())']
     """
-
 ````
 
 ```python
@@ -61,15 +45,29 @@ def truncate_number(number: float) -> float:
     >>> truncate_number(3.5)
     0.5
     """
-
 ````
 
-For each problem, instead of 200 candidate solutions, we will only generate 20 samples for illustration purposes. We use Nucleus sampling with `top-p=0.95` and `temperature=0.2`. For more details about decoding strategies for language generation, we recommend this [blog](https://huggingface.co/blog/how-to-generate). We will compute pass@1, pass@5 and pass@10, each correspending to unit test pass rate when selecting respectively 1, 5 and 10 samples from the candidate solutions.
+For each problem, instead of 200 candidate solutions, we will only generate 20 samples for illustration purposes. We use Nucleus sampling with `top-p=0.95` and `temperature=0.2`. For more details about decoding strategies for language generation, we recommend this [blog](https://huggingface.co/blog/how-to-generate). We will compute pass@1, pass@10 and pass@20, each correspending to unit test pass rate when selecting respectively 1, 10 and 20 samples from the candidate solutions.
 
 ```
 
-scores
+Results: {'pass@1': 0.0750, 'pass@10': 0.4473, 'pass@20': 0.5}
 
 ````
 
-If we take a closer look at the unit test results for each candidate solution in the three tasks, we find that only 3 passed the test which corresponds to `1/30 = 0.333`, our pass@1, the scores pass@5 and pass@10 are higher, because the more samples we select from the candidate solutions, the more likely we are to include the correct solution. Without surprise pass@10 is '2/3=0.73': if we select all candidates two tasks out of three get solved.
+If we take a closer look at the unit test results for each candidate solution in the three tasks, we find that only 3 passed the test for the second problem, and none did for the first problem. This means that we have 3 correct solutions among 40, which corresponds to our pass@1 value `3/40 = 0.075`. The scores pass@10 and pass@20 are higher, because the more samples we select from the candidate completions, the more likely we are to include the correct implementation. As
+for pass@20, it is '1/2=0.5', if we select all 20 candidates for each problem, the second problem get solved wich gives 50% success rate. If you are curious about the candidate solutions that passed the tests, they all implemented this function:
+
+```python
+
+def truncate_number(number: float) -> float:
+    """ Given a positive floating point number, it can be decomposed into
+    and integer part (largest integer smaller than given number) and decimals
+    (leftover part always smaller than 1).
+
+    Return the decimal part of the number.
+    >>> truncate_number(3.5)
+    0.5
+    """
+    return number % 1
+```