zhiminy/EvalEng huggingface dataset:
Repository: openai/human-eval
Issue: why use ThreadPoolExecutor with GIL in background?
Root cause: Architectural Constraint

In evaluation the code uses ThreadPoolExecutor at first and in each thread use multiprocessing package. Why not use ProcessPoolExecutor at first? Is there any consideration of optimizing performance?

@johnmclain 

The ThreadPoolExecutor  is used for concurrently executing the validation task for multiple generated code samples. Now these code sample validation are further wrapped as a process via multiprocessing to contain and isolate it . This ensures there is no conflict and also as a system security measure , executing and validating unsafe code.

Now why not processpool directly?

Threads are light and scales well with resources.
using a processpool directly is slightly less secure approach to validate unsafe code

If the code is safe code , using processpool directly makes a better sense.
Repository: openai/human-eval
Issue: Problems with installation instructions
Root cause: Documentation Deficiency

After setting up my environment with conda, I ran into problems with

```
$ git checkout https://github.com/openai/human-eval
$ pip install -e human-eval
```

First, `git checkout` should be a `git clone`.  

Then, when I run the pip command, I get 

```ERROR: human-eval is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).```

Thanks for catching `git checkout`. It's fixed now.

Hm.. I haven't seen that before. What python/pip version are you using? Could you try again with python 3.7 and the latest pip? I'm using

```
$ pip --version
pypip 21.1.3 ...
```

@jonpincus , you can simply run pip install -e . from the human-eval directory after cloning.

I keep getting this error:

```ModuleNotFoundError: No module named 'human_eval.data'; 'human_eval' is not a package```

> @jonpincus , you can simply run pip install -e . from the human-eval directory after cloning.

I think the **human-eval** here is the root path of this repo
@mtisz **human_eval** is a sub dir
Repository: openai/human-eval
Issue: Evaluations timing out
Root cause: Resource Mishandling

Hello,

I've been messing around with the benchmark for the last couple of days, everything as been running perfectly. All of a sudden the evaluations started to time out and I can't figure out why, been trying to find the issue the last few days. Anyone has come across something equal?

Better explanation:
After running the benchmark in generation+evaluation mode the generation works perfectly, all the samples look correct but every time the threads start to evaluate they always time out giving terrible results for all models. I'm running the evaluations with:

WizardLM/WizardCoder-33B-V1.1
BitsandBytes 4bit quantization
Temperature 0.2
Top_k 0
Top_p 0.95
Nr_samples 1
Batch size 1
Max_len_generation 1024
Repository: openai/human-eval
Issue: Task 145 makes no sense
Root cause: Documentation Deficiency

The prompt, canonical solution and tests for task 145 are:

```python
def order_by_points(nums):
    """
    Write a function which sorts the given list of integers
    in ascending order according to the sum of their digits.
    Note: if there are several items with similar sum of their digits,
    order them based on their index in original list.

    For example:
    >>> order_by_points([1, 11, -1, -11, -12]) == [-1, -11, 1, -12, 11]
    >>> order_by_points([]) == []
    """
    def digits_sum(n):
        neg = 1
        if n < 0: n, neg = -1 * n, -1 
        n = [int(i) for i in str(n)]
        n[0] = n[0] * neg
        return sum(n)
    
    return sorted(nums, key=digits_sum)


def check(candidate):
    # Check some simple cases
    assert candidate([1, 11, -1, -11, -12]) == [-1, -11, 1, -12, 11]
    assert candidate([1234,423,463,145,2,423,423,53,6,37,3457,3,56,0,46]) == [0, 2, 3, 6, 53, 423, 423, 423, 1234, 145, 37, 46, 56, 463, 3457]
    assert candidate([]) == []
    assert candidate([1, -11, -32, 43, 54, -98, 2, -3]) == [-3, -32, -98, -11, 1, 2, 43, 54]
    assert candidate([1,2,3,4,5,6,7,8,9,10,11]) == [1, 10, 2, 11, 3, 4, 5, 6, 7, 8, 9]
    assert candidate([0,6,6,-76,-21,23,4]) == [-76, -21, 0, 4, 23, 6, 6]

    # Check some edge cases that are easy to work out by hand.
    assert True, "This prints if this assert fails 2 (also good for debugging!)"
```

This makes no sense for negative inputs. For example, look at the first test:

```
assert candidate([1, 11, -1, -11, -12]) == [-1, -11, 1, -12, 11]
```

One reasonable interpretation for "digit sum" for a negative integer would be just the digit sum of the absolute value. Another would be the negative of the digit sum of the absolute value. But neither of those rules seem to be used here. Instead, if we look at the canonical solution, the interpretation seems to be that we should think of the minus sign as applying only to the *first* digit in the number, so a number like -186 breaks down as (-1, 8, 6) and has a digit sum of 13.

Theoretically it would be possible to infer this rule from the example given in the prompt, but it's still extremely difficult to guess the intended meaning here, even for a human. This may be a deliberate design choice, but I think it's worth flagging here so people can at least be aware of it.

I think I have found the same problem.
As a human, if I cannot get the standard answer, even with the example in the prompt, I can hardly know about the exact meaning about what I should do.

Did you check with [Hierarchical Prompting Taxonomy](https://paperswithcode.com/sota/code-generation-on-humaneval?tag_filter=760)?
Repository: openai/human-eval
Issue: Evaluation doesn't work on Windows
Root cause: Environment Incompatibility

After getting a score of 0 every time, I looked at the samples.jsonl_results.jsonl file and the result for each is this: "failed: module 'signal' has no attribute 'setitimer'" 

This seems like a Windows/Unix issue. 

Same issue here

For Windows, I replaced the `signal` module by the `threading` module on https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/execution.py#L90-L99 to
```
import threading
@contextlib.contextmanager
def time_limit(seconds: float):
    def signal_handler():
        raise TimeoutException("Timed out!")
    timer = threading.Timer(seconds, signal_handler)
    timer.start()
    try:
        yield
    finally:
        timer.cancel()
```

Above didn't work for me, just ran inside linux vm using wsl

same issue here. Every LLM I  load in LMStudio, and test against HumanEval via the script below, I get 0/0 with the failure being the same module not being found.

*HumanEval.py*:
```python
import os
import json
from human_eval.data import write_jsonl, read_problems
from human_eval.evaluation import evaluate_functional_correctness
from local_llm_client import client

def generate_one_completion(prompt):
    messages = [{"role": "user", "content": prompt}]
    response = client.chat_completion_create(messages)
    return response['choices'][0]['message']['content']

def generate_completions(problems, output_file):
    samples = []
    for task_id, problem in problems.items():
        prompt = problem["prompt"]
        completion = generate_one_completion(prompt)
        samples.append({"task_id": task_id, "completion": completion})
    
    write_jsonl(output_file, samples)

if __name__ == "__main__":
    problems = read_problems()
    output_file = "completions.jsonl"
    
    generate_completions(problems, output_file)
    
    results = evaluate_functional_correctness(output_file)
    print(json.dumps(results, indent=2))
```

*local_llm_client.py:*
```python
import requests
import json

class LocalLLMClient:
    def __init__(self, base_url="http://localhost:4445"):
        self.base_url = base_url

    def chat_completion_create(self, messages, temperature=0.7, max_tokens=-1, stream=False):
        url = f"{self.base_url}/v1/chat/completions"
        headers = {"Content-Type": "application/json"}
        data = {
            "model": "nxcode-cq-7b-orpo-q8_0",  # Adjust this to match your model name
            "messages": messages,
            "temperature": temperature,
            "max_tokens": max_tokens,
            "stream": stream
        }

        response = requests.post(url, headers=headers, json=data)
        response.raise_for_status()
        return response.json()

client = LocalLLMClient()
```
Repository: openai/human-eval
Issue: About how to use
Root cause: Documentation Deficiency

I want to use human-eval to get the correct code and program code generated by LLM.
I am able to build the environment, but do not know how to execute it (including selecting the LLM).
If you could please tell me how to do this?

you can identify which model you want to use and the logic you want to follow in the generate_one_completion module indicated in the readme.
Repository: openai/human-eval
Issue: I run the test program ，but why my “example_samples.jsonl_results.jsonl” file displayed “all passed” ？
Root cause: Documentation Deficiency

![image](https://github.com/user-attachments/assets/3a888ce4-1e0e-4569-9a5c-97f1aa23285b)


Then what should it have?
Repository: openai/human-eval
Issue: Question about the generate_one_completion
Root cause: Documentation Deficiency

Hey, I wanna ask one question about this 'generate_one_completion' in your usage part in the README file. What is that? Is that any Python function that has complete logic?

{"task_id": "Corresponding HumanEval task ID", "completion": "Completion only without the prompt"}
We provide example_problem.jsonl and example_solutions.jsonl under data to illustrate the format and help with debugging.

Here is a nearly functional example code (you just have to provide generate_one_completion to make it work) that saves generated completions to samples.jsonl.

@xupeng1910 the generate_one_completion function returns the model output or the completed code without the input prompt to the model. As you can see in the example_solutions.jsonl and example_problem.jsonl, the input "prompt" in example_problem.jsonl is "def return1():\n" and the "completion"s are in example_solutions.jsonl, these will be the predictions from your model which will then be appended with the prompt and will be tested by executing them. So the `generate_one_completion` takes in one prompt, calls the model and returns the model_prediction
Repository: openai/human-eval
Issue: I have the windows problem
Root cause: Environment Incompatibility

![image](https://github.com/user-attachments/assets/259e833e-b6e4-43cc-8612-c1f8d3c64852)
the os is windows and error like this
How to solve it. I reviewed the question18, but seems not the same question?


command line: evaluate_functional_correctness samples.jsonl

Same issue here.
Repository: openai/human-eval
Issue: Fix a IndentationError
Root cause: Validation Gap

## Error Log

I encountered a bug when running the command: python human_eval/execution.py

```
File "/data/humaneval/human-eval/human_eval/execution.py", line 59
    result.append("passed")
IndentationError: expected an indented block
```

## Fix it

<img width="355" alt="1734486214050" src="https://github.com/user-attachments/assets/23ed4e84-34df-48a9-bf6f-0fcb521b47e1" />


Uncommenting line 58 should resolve this issue.
Repository: openai/human-eval
Issue: sanity-check always returns pass@1: 0.0
Root cause: Algorithmic Error

When running evaluate_functional_correctness with the provided example files (data/example_samples.jsonl and data/example_problem.jsonl), the output always shows pass@1: 0.0. According to the documentation, it should be around 0.5.

Command: 
evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl

Result:
Reading samples...
6it [00:00, 2636.27it/s]
Running test suites...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00,  7.23it/s]
Writing results to data/example_samples.jsonl_results.jsonl...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 11915.64it/s]
{'pass@1': 0.0}

the same error
Repository: openai/human-eval
Issue: SETUP ERROR
Root cause: Validation Gap

When I try to run `pip install e .`. I have met the problem  

`ERROR: For req: human-eval==1.0. Invalid script entry point: <ExportEntry evaluate_functional_correctness = human_eval.evaluate_functional_correctness:None []> - A callable suffix is required. Cf https://packaging.python.org/specifications/entry-points/#use-for-scripts for more information.`

CHANGE THE `[setup.py](https://github.com/openai/human-eval/blob/master/setup.py)` to 

``` python
import os

import pkg_resources
from setuptools import setup, find_packages

setup(
    name="human-eval",
    version="1.0",
    description="",
    author="OpenAI",
    packages=find_packages(),
    install_requires=[
        str(r)
        for r in pkg_resources.parse_requirements(
            open(os.path.join(os.path.dirname(__file__), "requirements.txt"))
        )
    ],
    entry_points={
        "console_scripts": [
            "evaluate_functional_correctness = human_eval.evaluate_functional_correctness:main",
        ]
    },
)
```

Then, it will not throw out error to me

Thanks man! It worked for me as well

thank you very much

thank you very much！！！
Repository: openai/human-eval
Issue: How example_samples.jsonl and example_problem.jsonl are created?
Root cause: Documentation Deficiency

Hi,
I was wondering how the example_samples.jsonl and example_problem.jsonl files are created? What is the logic behind them?
I assume the example_samples.jsonl is the openai/openai_humaneval dataset with all of its columns there, but how example_problem.jsonl should be created for it?
Thanks!
Repository: openai/human-eval
Issue: Error in the prompt of HumanEval/47
Root cause: Validation Gap

```python
def median(l: list):
    """Return median of elements in the list l.
    >>> median([3, 1, 2, 4, 5])
    3
    >>> median([-10, 4, 6, 1000, 10, 20])
    15.0
    """
```

Maybe the output of `median([-10, 4, 6, 1000, 10, 20])` should be `8` not `15.0` Can you please check it?

Here's the test with `statistics` library

```python
>>> import statistics
>>> statistics.median([-10, 4, 6, 1000, 10, 20])
8.0
```

I've found the same problem.
However, the question is: Is this an error in the annotation process that needs to be fixed, or a feature intended to test the completion performance under imperfect prompts?
Repository: openai/human-eval
Issue: Function Name Mismatch in Prompt vs. Test Code – HumanEval/161
Root cause: Documentation Deficiency

I’ve discovered a bug in the HumanEval dataset, specifically in `Task ID: HumanEval/161`, which causes incorrect evaluation of LLM-generated code.

In the prompt for this task, the function name is defined as `solve`, as shown below:

```
def solve(s):
    """You are given a string s.
    if s[i] is a letter, reverse its case from lower to upper or vise versa, 
    otherwise keep it as it is.
    If the string contains no letters, reverse the string.
    The function should return the resulted string.
    Examples
    solve("1234") = "4321"
    solve("ab") = "AB"
    solve("#a@C") = "#A@c"
    """
```

However, the test code for the same task calls a function named `candidate`:

```
def check(candidate):
    # Check some simple cases
    assert candidate("AsDf") == "aSdF"
    assert candidate("1234") == "4321"
    assert candidate("ab") == "AB"
    assert candidate("#a@C") == "#A@c"
    assert candidate("#AsdfW^45") == "#aSDFw^45"
    assert candidate("#6@2") == "2@6#"

    # Check some edge cases that are easy to work out by hand.
    assert candidate("#$a^D") == "#$A^d"
    assert candidate("#ccc") == "#CCC"

    # Don't remove this line:
    ...
```

Because LLMs typically generate a function named solve (as instructed by the prompt), this discrepancy leads to false negatives in test results. The generated function may be logically correct, but it fails the tests due to the incorrect function name.

**Why this matters:**
This inconsistency causes misleading evaluation outcomes and can unfairly penalize model performance.

**Suggested Resolution:**
Ensure the function name in the prompt examples matches the function name expected by the test code (`candidate`), or vice versa — ideally using a consistent and correct name like solve across both.

### Additional Issue: Test Code is Not Executed Properly

Beyond the function name mismatch, I also noticed that the `check()` function in the test code is defined but never called. This causes an even more serious issue during evaluation: the `assert` statements inside `check()` are never executed.

This issue appears to be an incorrect AI hallucination and should be closed.

If you look at the test harness, you can see that the test program is built by pasting the `entry_point`, which is `solve` in our case, into the `check` function call:

https://github.com/openai/human-eval/blob/6d43fb980f9fee3c892a914eda09951f772ad10d/human_eval/execution.py#L33

Or in other words, the following is appended to the test program:

```python
check(solve)
```
Repository: openai/human-eval
Issue: Fix invalid console_scripts entry point format in setup.py
Root cause: Validation Gap


```markdown
## Issue Description

The `console_scripts` entry point in `setup.py` uses an invalid format that causes installation failures with strict package managers like `uv` (and may break CLI usage in other tools).

### Current (incorrect) format:
```python
"evaluate_functional_correctness = human_eval.evaluate_functional_correctness"
```

### Correct format (per [PEP 452](https://peps.python.org/pep-0452/) / [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#console-scripts)):
```python
"evaluate_functional_correctness = human_eval.evaluate_functional_correctness:main"
```

## Impact

- Tools like [`uv`](https://github.com/astral-sh/uv) **fail to install** the package with error:
  ```
  error: The wheel is invalid: invalid console script: 'human_eval.evaluate_functional_correctness'
  ```
- Even with `pip`, the CLI command `evaluate_functional_correctness` may not work correctly because the entry point lacks the `:main` function reference.

## Suggested Fix

Update `setup.py` line:
https://github.com/openai/human-eval/blob/7faa4b7/setup.py#L33

From:
```python
"evaluate_functional_correctness = human_eval.evaluate_functional_correctness",
```

To:
```python
"evaluate_functional_correctness = human_eval.evaluate_functional_correctness:main",
```

This is a one-line fix that improves compatibility with modern Python tooling.

---

> Note: I understand this repository is archived, but this fix would help users who depend on `human-eval` via Git URLs in `pyproject.toml` (e.g., with `uv`, `pip`, or `poetry`).
```
Repository: openai/human-eval
Issue: Unsafe execution of untrusted code via `exec` by default
Root cause: Validation Gap

The library directly calls `exec()` directly on model-generated code. The warning notes this should be commented out by default, but it isn't.

https://github.com/openai/human-eval/blob/6d43fb980f9fee3c892a914eda09951f772ad10d/human_eval/execution.py#L40-L50

The best solution would be to provide at least one sandbox integration as a reasonable default. I'm open to contributing this.

@mpokrass
Repository: openai/human-eval
Issue: Re-produce raw GPT-Neo with 125M and 1.3B on this human-eval dataset

Hi, we reproduced the performance of the raw GPT-Neo (125M and 1.3B) on the HumanEval dataset, and found that it was much lower than that reported in the Codex paper. Do you have any plans to publish the raw GPT-Neo on HumanEval? In addition, are there any tricks in the process of reproducing this? Thanks!

Our re-produce results:
![image](https://user-images.githubusercontent.com/84262181/142711224-c9bb29c9-ff08-4da9-8356-b9ec678ca54a.png)

The official reported results:
![image](https://user-images.githubusercontent.com/84262181/142711267-7eec5471-2ed4-47e0-b788-a99848a50e8c.png)

Looking forward to your reply!

Hi BitcoinNLPer, 
Have you fixed this problem? If yes, would it be possible to explain problems and give the solution?

Thank you so much.
Repository: plurai-ai/intellagent
Issue: Issue on checking the Plurai-ai intellagent tool
Root cause: Validation Gap

After running the application, an error occurs when selecting events to be visualized. The issue seems to be related to Streamlit’s session state and database query execution.

Steps to Reproduce:
Run the application successfully.
Select an event/thread to visualize.
The application encounters an error and fails to load the expected data.
Observed Behavior:
The error traceback shows that the issue arises within Streamlit’s session state management when executing a callback function.

The error occurs in the function on_select_thread, which attempts to execute the following query:

cursor.execute("SELECT * FROM Dialog WHERE thread_id = ? ORDER BY time ASC", (thread_id,))
The following traceback is observed:

File "...streamlit\runtime\scriptrunner\exec_code.py", line 88, in exec_func_with_error_handling  
    result = func()  
File "...streamlit\runtime\scriptrunner\script_runner.py", line 541, in code_to_exec  
    self._session_state.on_script_will_rerun(  
File "...streamlit\runtime\state\safe_session_state.py", line 66, in on_script_will_rerun  
    self._state.on_script_will_rerun(latest_widget_states)  
File "...streamlit\runtime\state\session_state.py", line 559, in on_script_will_rerun  
    self._call_callbacks()  
File "...streamlit\runtime\state\session_state.py", line 572, in _call_callbacks  
    self._new_widget_state.call_callback(wid)  
File "...streamlit\runtime\state\session_state.py", line 273, in call_callback  
    callback(*args, **kwargs)  
File "......NewToolByGitHub_Gan\intellagent-main\simulator\visualization\pages\2_🤖_Session_Visualizer.py", line 80, in on_select_thread  
    cursor.execute("SELECT * FROM Dialog WHERE thread_id = ? ORDER BY time ASC", (thread_id,))  

![Image](https://github.com/user-attachments/assets/105e45a0-2682-419b-a075-a3bb01443918)

Hi, It looks like you had a failure during the simulator execution and none of the rollout was completed.
When you are looking at the first page with the analysis are you seeing something?
You can also look at the experiment folder in the log file to see if there was an issue in executing one of the chains