zhiminy/EvalEng huggingface dataset:
Repository: openai/evals
Issue: Remove leading whitespace from sampled answers in Match template
Root cause: Validation Gap

### Describe the feature or improvement you're requesting

The models below gpt-3.5-turbo (text-davinci-003, text-curie-001, etc.) seem to frequently begin their sampled responses with whitespace (spaces or newlines). This causes the Match eval template to fail, as in this example, where `" m3"` does not match with `"m3"`:

```
{"run_id": "2305070508162VM5L4DR", "event_id": 6, "sample_id": "note-intervals.dev.6", "type": "sampling", "data": {"prompt": "You are a note interval calculator. Given two notes separated by a space, you will calculate the interval between them. Respond only with the abbreviated interval name (e.g. P4, d5).\nUser: C Eb\nAssistant: ", "sampled": [" m3"]}, "created_by": "", "created_at": "2023-05-07 05:08:17.659629+00:00"}
{"run_id": "2305070508162VM5L4DR", "event_id": 7, "sample_id": "note-intervals.dev.6", "type": "match", "data": {"correct": false, "expected": "m3", "picked": null, "sampled": " m3", "options": ["m3"]}, "created_by": "", "created_at": "2023-05-07 05:08:17.659629+00:00"}
```

However if we do not use the Match template, and instead use Includes or FuzzyMatch, this introduces false positives for pure matching tasks. 

For Includes, we can get that `"A4"` matches witih `"A2"`:

```
{"run_id": "230509162956UNZNCLEW", "event_id": 0, "sample_id": "note-intervals.dev.5", "type": "sampling", "data": {"prompt": "What is the interval between the following two notes? Respond only with the abbreviated interval name (e.g. P4, d5).\nUser: C D#\nAssistant: ", "sampled": ["A4"]}, "created_by": "", "created_at": "2023-05-09 16:29:57.543580+00:00"}
{"run_id": "230509162956UNZNCLEW", "event_id": 1, "sample_id": "note-intervals.dev.5", "type": "metrics", "data": {"accuracy": 1.0}, "created_by": "", "created_at": "2023-05-09 16:29:57.543580+00:00"}
```

And for FuzzyMatch, we can get that `"A#1"` matches with `"A1"`:

```
{"run_id": "230509010553PJ2ND4CN", "event_id": 9, "sample_id": "note-intervals.dev.1", "type": "sampling", "data": {"prompt": "What is the interval between the following two notes? Respond only with the abbreviated interval name (e.g. P4, d5).\nUser: C C#\nAssistant: ", "sampled": [" A#1"]}, "created_by": "", "created_at": "2023-05-09 01:05:54.810508+00:00"}
{"run_id": "230509010553PJ2ND4CN", "event_id": 10, "sample_id": "note-intervals.dev.1", "type": "match", "data": {"correct": true, "expected": "A1", "picked": [" A#1"]}, "created_by": "", "created_at": "2023-05-09 01:05:54.810508+00:00"}
{"run_id": "230509010553PJ2ND4CN", "event_id": 11, "sample_id": "note-intervals.dev.1", "type": "metrics", "data": {"accuracy": 1.0, "f1_score": 1.0}, "created_by": "", "created_at": "2023-05-09 01:05:54.810508+00:00"}
```

So for pure matching tasks, I believe it makes sense to remove leading whitespace from the sampled answers, and would only improve eval correctness. 

### Additional context

_No response_
Repository: openai/evals
Issue: Is there a list of tasks that chatgpt fails?

Hi, is it possible to see a collection of questions that gpt-4 is failing? I wan't to test some prompts I wrote that improve accuracy, and I thought it would be good to focus on tasks that it is currently failing. Thanks!

try the logic evals  https://github.com/openai/evals/tree/main/evals/registry/data/logic

They fail even with cot of reasoning
Repository: openai/evals
Issue: RateLimitError Causes Duplicate Logs and Incorrect Metrics
Root cause: Algorithmic Error

### Describe the bug

If a RateLimit error occurs when running an eval, the evaluation code may run a sample more than once, causing the final reported accuracy to be incorrect. 

In my case, a single log file contained these two results (exact same sample_id, but different timestamps indicating different runs):

```
{"run_id": "230510185932RNI3M7IV", "event_id": 184, "sample_id": "note-intervals.dev.316", "type": "sampling", "data": {"prompt": [{"role": "system", "content": "You are a note interval calculator. Given two notes, calculate the interval between them. Respond only with the abbreviated interval name (e.g. P4, d5)."}, {"role": "user", "content": "D# F"}], "sampled": ["m3"]}, "created_by": "", "created_at": "2023-05-10 19:02:26.613346+00:00"}
{"run_id": "230510185932RNI3M7IV", "event_id": 185, "sample_id": "note-intervals.dev.316", "type": "match", "data": {"correct": false, "expected": "d3", "picked": null, "sampled": "m3", "options": ["d3"]}, "created_by": "", "created_at": "2023-05-10 19:02:26.613346+00:00"}
```
```
{"run_id": "230510185932RNI3M7IV", "event_id": 194, "sample_id": "note-intervals.dev.316", "type": "sampling", "data": {"prompt": [{"role": "system", "content": "You are a note interval calculator. Given two notes, calculate the interval between them. Respond only with the abbreviated interval name (e.g. P4, d5)."}, {"role": "user", "content": "D# F"}], "sampled": ["m3"]}, "created_by": "", "created_at": "2023-05-10 19:02:46.433398+00:00"}
{"run_id": "230510185932RNI3M7IV", "event_id": 195, "sample_id": "note-intervals.dev.316", "type": "match", "data": {"correct": false, "expected": "d3", "picked": null, "sampled": "m3", "options": ["d3"]}, "created_by": "", "created_at": "2023-05-10 19:02:46.433398+00:00"}
```

This caused the final accuracy to be misreported in both the terminal output and log file as 93/433=0.21478 instead of 93/432=0.21527 (there are only 432 samples in the samples.jsonl file):
```
[2023-05-10 15:10:24,894] [oaieval.py:149] accuracy: 0.21478060046189376
```

Obviously this is a minor problem, but is clearly a correctness issue. As far as I can tell hitting a RateLimitError should only cause the requests to back of temporarily and then continue later, and should not indicate an eval failure. I am not familiar enough with the code to suggest a fix.

### To Reproduce

Reproducing this issue will depend on your rate limits, but simply find and run a sufficiently large eval such that you observe an openai.error.RateLimitError. Read the log file and check for duplicate sample_ids.

### Code snippets

_No response_

### OS

Windows 10

### Python version

v3.9.16

### Library version

Cloned from source

https://github.com/openai/evals/pull/987 should fix the issue, closing for now. Let me know if you see it again with the latest changes.
Repository: openai/evals
Issue: eval approach:  GPT4 provides alternative responses with reasoning and then has to pick the best one
Root cause: Unimplemented Feature Gap

### Describe the feature or improvement you're requesting

I think it would be interesting to see how GPT4 performs on more ideal scenarios, where COT  is allowed extensively.  

In particular, GPT4 is allowed to think step by step along several different approaches to solve a problem and provide its reasoning.  Some prompting should be done to ensure that it tries to look at the problem from different angles / viewpoints / perspectives.

Then, in a second prompt, it's asked to pick from the approach it thinks is most likely to be correct.  

If there has been a paper or effort done on evaluating GPT4 for something like this, I'd greatly appreciate a link.  :)

### Additional context

_No response_

https://arxiv.org/pdf/2305.10601.pdf
Repository: openai/evals
Issue: wrong information in Estonian about the product
GitHub labels: bug

### Describe the bug

Product name is Pedibus forte. Chat GPT gives information that it is against urinal inflammation and writes the wrong compositions. Actually, this product helps with fungal infections.

### To Reproduce

chat GPT has to ask what active ingredients are in the product or just ingredients. So, in this case, it can give the right information about products, drugs, cosmetics etc. 
Sometimes the problem of understanding the specification of products is the product name. Depending on countries product with the same ingredients has different names.  

### Code snippets

_No response_

### OS

windows 

### Python version

Python

### Library version

GPT-3.5

Please report this via ChatGPT's feedback tool.
Repository: openai/human-eval
Issue: Will this be helpful for people reading the paper ? 

I was reading your [paper](https://arxiv.org/pdf/2107.03374.pdf). And was interested in seeing what the test data looks like. 
But you have given it in ```.jsonl``` format which is okay if you are doing a careful analysis of the test data. 

But for quick readers parsing the test data requires ```willingness``` and they might not want to do it. 

So I parsed the ```data.jsonl``` into Human Readable Prompt and Canoncial Sections : https://gist.github.com/jalotra/f4d119e8108fcd44b932103d73140381. 

Will this add value to this repo if we can add this for ```a quick read``` ?  

I agree, two or three example datapoints in the README for this repo would be really helpful.

Great sharing!
Repository: openai/human-eval
Issue: Evaluation.py failing on KeyError: 'test/0'
Root cause: Validation Gap

I tried running:

evaluate_functional_correctness ./data/example_samples.jsonl 

Getting the following error:

 ```
File "/Users/brianriviere/projects/human-eval/human_eval/evaluation.py", line 65, in evaluate_functional_correctness
    args = (problems[task_id], completion, timeout, completion_id[task_id])
KeyError: 'test/0'
```

Is there something I'm not doing correctly?


What about "evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl"?

EOFError

thanks Henry, your reply is really helpful
Repository: openai/human-eval
Issue: Prompt used in APPS
Root cause: Documentation Deficiency

Thank you for the very interesting work! 

I have one question about the natural language prompt used in APPS. 

Did you directly use the original prompts used in the APPS benchmark? (as coded [here](https://github.com/hendrycks/apps/blob/main/eval/generate_gpt_codes.py) in the `generate_prompt` function). 

You mentioned in the paper that 'we append a single input/output example from the task description to the docstring as a formatting hint.' How did you do this precisely? Did you need to construct a new prompt, including a function signature and a docstring with input/output example? 




We constructed a new prompt, below is the exact prompt we used. We did not tune it in any special way, so I suspect other prompts would work too. Hope that helps!

```
# Language: Python 3
# Task: Synthesize program

"""
Contains programming exercises for single functions specified by their
doc-strings and with solutions in simple code and with a lot of comments
that explain what is done and why and how it is related to the specification.
"""

# Example 1.

"""
Given an array of integers, find if the array contains any duplicates.

Your function should return true if any value appears at least twice in the array, and
 it should return false if every element is distinct.

-----Input-----

The first line contains a list of integers.


-----Output-----

Output true if there are duplicates and false otherwise.

-----Examples-----
Input
[1,2,3,1]

Output
true

Input
[1,2,3,4]

Output
false

Input
[1,1,1,3,3,4,3,2,4,2]

Output
true

"""

# Solution:
list = eval(input())
if len(list) == len(set(list)):
    return False
else:
    return True

# Example 2.

```

Thank you for your answer!
Repository: openai/human-eval
Issue: Entry point error while Installing the package.
Root cause: Validation Gap

Hi,

I am having trouble installing the package. 

Installation Command: `pip install git+http://github.com/openai/human-eval#egg=human-eval` 

Error: `For req: human-eval. Invalid script entry point: <ExportEntry evaluate_functional_correctness = human_eval.evaluate_functional_correctness:None []>`

Possible Solution:  Manually edit the setup.py file and change the entry point to the main function instead of None.

Is it possible to include this change in the new version?

is this fixed? I don't see `setup.py` being modified and bumped into this same error while doing a `pip install -e human-eval` on macOS within a virtual environment

hello,
some weird things going on here! A couple of days ago installing and testing a fork with the following code was fine
```
$ git clone https://github.com/kingabzpro/human-eval
$ pip install -e human-eval
$ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl
```
Today impossible to install on a brand new env with the following error
```
error: Failed to install: human_eval-1.0-py3-none-any.whl (human-eval==1.0 (from file:///home/ubuntu/test-humaneval/human-eval))
  Caused by: The wheel is invalid: invalid console script: 'evaluate_functional_correctness'
```
I finally ended up modifying the `setup.py` by specifying the function in the entry point
```
  entry_points={
        "console_scripts": [
            "evaluate_functional_correctness = human_eval.evaluate_functional_correctness:main",
        ]
    }
```
and only then the code installed correctly 🤔

@sdelahaies You are my hero
Repository: openai/human-eval
Issue: pass@k on filtered samples  
Root cause: Documentation Deficiency

Hi, 

Thank you for the great work! 

I have 2 questions about the computation of the pass@k metric after applying filtering on the APPS benchmark.

1. Will the `total` array in the below code snippet contain numbers of filtered samples that passed the example test cases (from problem statement), i.e. each number <= N_original_samples(=1000)? 
https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/evaluation.py#L85

2. In the cases when a number of filtered samples is less than k (=[1,5]), how do you compute the pass@k metric for these cases? For example, when N_filtered_samples = 1 and k=5, can we assume execution results of 4 failures and 1 passed/failure (depending on the final unit test results of this filtered sample)?
Repository: openai/human-eval
Issue: Why not allow contribution?

I am curious as to why this data set is not open for contribution to keep it evolving. Yes, ["164 hand-written programming problems"](https://arxiv.org/abs/2107.03374) is a good start, but more is certainly better, especially that all the problems seems to be focusing on algorithms. By opening this for contribution, you crowd source this problem. Obviously, contributions have to meet a certain standard to avoid degrading the quality of the data set, but this is not hard to achieve.

Another problem that might happen from allowing contribution is it might make hard to reference in papers, but surely a simple versioning system can solve this problem. Authors can then say something like: We achieved X% accuracy on HumanEval version Y.
Repository: openai/human-eval
Issue: Why pass@k =1.0? use the "evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl"
Root cause: Validation Gap

$ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl
Reading samples...
6it [00:00, 7047.28it/s]
Running test suites...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 98.99it/s]
Writing results to data/example_samples.jsonl_results.jsonl...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 21826.39it/s]
{'pass@1': 1.0}

It looks like you omitted exec line  @Smithol 

Do you know what is wrong with it?

> $ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl Reading samples... 6it [00:00, 7047.28it/s] Running test suites... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 98.99it/s] Writing results to data/example_samples.jsonl_results.jsonl... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 21826.39it/s] {'pass@1': 1.0}

After uncomment execution.py --->line 58--->exec() function I get 0.5.
Repository: openai/human-eval
Issue: Error in canonical solution and tests for HumanEval/163
Root cause: Documentation Deficiency

The following solution and tests don't seem to match the description. Can you explain why we are bounding lower and upper to be between 2 and 8?

`def generate_integers(a, b):
    """
    Given two positive integers a and b, return the even digits between a
    and b, in ascending order.

    For example:
    generate_integers(2, 8) => [2, 4, 6, 8]
    generate_integers(8, 2) => [2, 4, 6, 8]
    generate_integers(10, 14) => []
    """

    lower = max(2, min(a, b))
    upper = min(8, max(a, b))

    return [i for i in range(lower, upper+1) if i % 2 == 0]

def check(candidate):

    # Check some simple cases
    assert candidate(2, 10) == [2, 4, 6, 8], "Test 1"
    assert candidate(10, 2) == [2, 4, 6, 8], "Test 2"
    assert candidate(132, 2) == [2, 4, 6, 8], "Test 3"
    assert candidate(17,89) == [], "Test 4"`

The natural language description in the docstring requires the program to find `digits` instead of `numbers`, which means `a` and `b` must be in `[0, 9]`, and 2 and 8 are the largest and smallest digits. I think maybe this is why.


> Given two positive integers a and b, return the even digits between a
> and b, in ascending order.

It should be
```
Given two positive integers a and b, return the single even digit integers between a
and b, in ascending order.
```

The phrase "digits between integers" is inherently meaningless.
Repository: openai/human-eval
Issue: Error in canonical solution 95 check_dict_case
Root cause: Algorithmic Error


```python
def check_dict_case(dict: Dict[str, str]) -> bool:
    """
    Given a dictionary, return True if all keys are strings in lower 
    case or all keys are strings in upper case, else return False.
    The function should return False is the given dictionary is empty.
    Examples:
    # >>> check_dict_case({"a":"apple", "b":"banana"})
    # True
    # >>> check_dict_case({"a":"apple", "A":"banana", "B":"banana"})
    # False
    # >>> check_dict_case({"a":"apple", 8:"banana", "a":"apple"})
    # False
    # >>> check_dict_case({"Name":"John", "Age":"36", "City":"Houston"})
    # False
    # >>> check_dict_case({"STATE":"NC", "ZIP":"12345" })
    True
    """
    if len(dict.keys()) == 0:
        return False
    else:
        state = "start"
        for key in dict.keys():

            if isinstance(key, str) == False:
                state = "mixed"
                break
            if state == "start":
                if key.isupper():
                    state = "upper"
                elif key.islower():
                    state = "lower"
                else:
                    break
            elif (state == "upper" and not key.isupper()) or (state == "lower" and not key.islower()):
                    state = "mixed"
                    break
            else:
                break
        return state == "upper" or state == "lower"
```

In the solution above, the last `break` statement should instead be `continue`

In the current test 
```python
assert candidate({"p":"pineapple", "A":"banana", "B":"banana"}) == False
```
current solution would work because the case switch occurs in the second element, and that python happens to preserve the order in which the key, value pairs are iterated. 

If the test were changed to
```python
assert candidate({"A":"banana", "B":"banana", "p":"pineapple"}) == False
```

This solution would have failed the test
Repository: openai/human-eval
Issue: # Realizar ajustes en la arquitectura 


# Realizar ajustes en la arquitectura o hiperparámetros del modelo
modelo.add(tf.keras.layers.Dropout(0.2))
modelo.add(tf.keras.layers.Dense(units=16, activation='relu'))

# Volver a entrenar el modelo con los ajustes realizados
modelo.fit(datos_entrenamiento_preprocesados, etiquetas_entrenamiento, epochs=5, batch_size=32)
Repository: openai/human-eval
Issue: execution.py bug request
Root cause: Validation Gap

bug request for execution.py

This part (https://github.com/openai/human-eval/blob/77b90b8f70e2553ba720c3d24156acfd28104ec4/human_eval/execution.py#L48)

```
            try:
                exec_globals = {}
                with swallow_io():
                    with time_limit(timeout):
# WARNING
# This program exists to execute untrusted model-generated code. Although
# it is highly unlikely that model-generated code will do something overtly
# malicious in response to this test suite, model-generated code may act
# destructively due to a lack of model capability or alignment.
# Users are strongly encouraged to sandbox this evaluation suite so that it 
# does not perform destructive actions on their host or network. For more 
# information on how OpenAI sandboxes its code, see the accompanying paper.
# Once you have read this disclaimer and taken appropriate precautions, 
# uncomment the following line and proceed at your own risk:
#                         exec(check_program, exec_globals)
                result.append("passed")
            except TimeoutException:
                result.append("timed out")
            except BaseException as e:
                result.append(f"failed: {e}")

            # Needed for cleaning up.
            shutil.rmtree = rmtree
            os.rmdir = rmdir
            os.chdir = chdir

```

should be fixed since it causes an error due to the indentation issue.

```
# WARNING
# ...
# Once you have read this disclaimer and taken appropriate precautions, 
# uncomment the following line and proceed at your own risk:
#                         exec(check_program, exec_globals)
```

You need to uncomment the above line. Also, please read our instructions/warning more carefully to understand what the reliability guard can and can not do.
Repository: openai/human-eval
Issue: Why do I use the phi model to output the same result for all samples at a temperature of 0.8?
Root cause: Validation Gap

def generate_one_completion(prompt: str):
    torch.set_default_device("cuda")
    model = AutoModelForCausalLM.from_pretrained("/****/phi-1", trust_remote_code=True)
    tokenizer = AutoTokenizer.from_pretrained("/****/phi-1", trust_remote_code=True)
    # inputs = tokenizer("'''"+prompt+"'''", return_tensors="pt", return_attention_mask=False)
    inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
    # outputs = model.generate(**inputs, max_length=200,max_new_tokens=430)
    outputs = model.generate(**inputs, max_length=200,temperature=0.8,do_sample=True)
    completion = tokenizer.batch_decode(outputs)[0]
    return completion

This is my model output code. Regardless of the value of num_samples_per_task set, it returns the same answer for each question
Repository: openai/human-eval
Issue: bug in estimate_pass_at_k
Root cause: Algorithmic Error

https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/evaluation.py#L26

This code returns 1 when c=0 and n < k, whereas 0 is expected.