Repository: evalplus/evalplus
Issue: Where are the final model outputs used to calculate `pass@1`?
GitHub labels: help wanted

Love your project! 🚀 

Where can I find the final model outputs used to calculate `pass@1` with EvalPlus?

I could only reproduce the results **without** EvalPlus by analyzing the files you made available [in this release](https://github.com/evalplus/evalplus/releases/tag/v0.1.0).

<details>
  <summary>I have written my own code to calculate `pass@1` on these model outputs
</summary>

  ```python
  import multiprocessing
import requests
import zipfile
import io
from tqdm import tqdm
from evaluate import load
from datasets import load_dataset
import os


os.environ["HF_ALLOW_CODE_EVAL"] = "1"
DOWNLOAD = False
results_folder = "./JawTitan/EvalPlus/humaneval/"

if __name__ == '__main__':

    files = ['chatgpt_temp_0.0.zip', 'chatgpt_temp_0.8.zip', 'code-llama-13b_temp_0.0.zip',
             'code-llama-13b_temp_0.2.zip',
             'code-llama-13b_temp_0.4.zip', 'code-llama-13b_temp_0.6.zip', 'code-llama-13b_temp_0.8.zip',
             'code-llama-34b_temp_0.0.zip', 'code-llama-34b_temp_0.2.zip', 'code-llama-34b_temp_0.4.zip',
             'code-llama-34b_temp_0.6.zip', 'code-llama-34b_temp_0.8.zip', 'code-llama-7b_temp_0.0.zip',
             'code-llama-7b_temp_0.2.zip', 'code-llama-7b_temp_0.4.zip', 'code-llama-7b_temp_0.6.zip',
             'code-llama-7b_temp_0.8.zip', 'codegen-16b_temp_0.0.zip', 'codegen-16b_temp_0.2.zip',
             'codegen-16b_temp_0.4.zip', 'codegen-16b_temp_0.6.zip', 'codegen-16b_temp_0.8.zip',
             'codegen-2b_temp_0.0.zip',
             'codegen-2b_temp_0.2.zip', 'codegen-2b_temp_0.4.zip', 'codegen-2b_temp_0.6.zip', 'codegen-2b_temp_0.8.zip',
             'codegen-6b_temp_0.0.zip', 'codegen-6b_temp_0.2.zip', 'codegen-6b_temp_0.4.zip', 'codegen-6b_temp_0.6.zip',
             'codegen-6b_temp_0.8.zip', 'codegen2-16b_temp_0.0.zip', 'codegen2-1b_temp_0.0.zip',
             'codegen2-1b_temp_0.2.zip',
             'codegen2-1b_temp_0.4.zip', 'codegen2-1b_temp_0.6.zip', 'codegen2-1b_temp_0.8.zip',
             'codegen2-3b_temp_0.0.zip',
             'codegen2-3b_temp_0.2.zip', 'codegen2-3b_temp_0.4.zip', 'codegen2-3b_temp_0.6.zip',
             'codegen2-3b_temp_0.8.zip',
             'codegen2-7b_temp_0.0.zip', 'codegen2-7b_temp_0.2.zip', 'codegen2-7b_temp_0.4.zip',
             'codegen2-7b_temp_0.6.zip',
             'codegen2-7b_temp_0.8.zip', 'codet5p-16b_temp_0.0.zip', 'codet5p-16b_temp_0.2.zip',
             'codet5p-16b_temp_0.4.zip',
             'codet5p-16b_temp_0.6.zip', 'codet5p-16b_temp_0.8.zip', 'codet5p-2b_temp_0.0.zip',
             'codet5p-6b_temp_0.0.zip',
             'gpt-4_temp_0.0.zip', 'gpt-j_temp_0.0.zip', 'gpt-j_temp_0.2.zip', 'gpt-j_temp_0.4.zip',
             'gpt-j_temp_0.6.zip',
             'gpt-j_temp_0.8.zip', 'gptneo-2b_temp_0.0.zip', 'gptneo-2b_temp_0.2.zip', 'gptneo-2b_temp_0.4.zip',
             'gptneo-2b_temp_0.6.zip', 'gptneo-2b_temp_0.8.zip', 'HumanEvalPlusGT.zip', 'incoder-1b_temp_0.0.zip',
             'incoder-1b_temp_0.2.zip', 'incoder-1b_temp_0.4.zip', 'incoder-1b_temp_0.6.zip', 'incoder-1b_temp_0.8.zip',
             'incoder-6b_temp_0.0.zip', 'incoder-6b_temp_0.2.zip', 'incoder-6b_temp_0.4.zip', 'incoder-6b_temp_0.6.zip',
             'incoder-6b_temp_0.8.zip', 'mistral-7b_temp_0.0.zip', 'mistral-7b_temp_0.2.zip', 'mistral-7b_temp_0.4.zip',
             'mistral-7b_temp_0.6.zip', 'mistral-7b_temp_0.8.zip', 'phind-code-llama-34b-v2_temp_0.0.zip',
             'phind-code-llama-34b-v2_temp_0.2.zip', 'phind-code-llama-34b-v2_temp_0.4.zip',
             'phind-code-llama-34b-v2_temp_0.6.zip', 'phind-code-llama-34b-v2_temp_0.8.zip', 'polycoder_temp_0.0.zip',
             'polycoder_temp_0.2.zip', 'polycoder_temp_0.4.zip', 'polycoder_temp_0.6.zip', 'polycoder_temp_0.8.zip',
             'santacoder_temp_0.0.zip', 'santacoder_temp_0.2.zip', 'santacoder_temp_0.4.zip', 'santacoder_temp_0.6.zip',
             'santacoder_temp_0.8.zip', 'stablelm-7b_temp_0.0.zip', 'stablelm-7b_temp_0.2.zip',
             'stablelm-7b_temp_0.4.zip',
             'stablelm-7b_temp_0.6.zip', 'stablelm-7b_temp_0.8.zip', 'starcoderbase-1b_temp_0.0.zip',
             'starcoderbase-3b_temp_0.0.zip', 'starcoderbase-7b_temp_0.0.zip', 'starcoder_temp_0.0.zip',
             'starcoder_temp_0.2.zip', 'starcoder_temp_0.4.zip', 'starcoder_temp_0.6.zip', 'starcoder_temp_0.8.zip',
             'vicuna-13b_temp_0.0.zip', 'vicuna-13b_temp_0.2.zip', 'vicuna-13b_temp_0.4.zip', 'vicuna-13b_temp_0.6.zip',
             'vicuna-13b_temp_0.8.zip', 'vicuna-7b_temp_0.0.zip', 'vicuna-7b_temp_0.2.zip', 'vicuna-7b_temp_0.4.zip',
             'vicuna-7b_temp_0.6.zip', 'vicuna-7b_temp_0.8.zip']

    if DOWNLOAD:
        for f in tqdm(files):

            if f[:-len(".zip")] in os.listdir(results_folder):
                continue

            # URL of the zip file
            url = f"https://github.com/evalplus/evalplus/releases/download/v0.1.0/{f}"

            # Send a GET request to the URL
            response = requests.get(url)

            if response.status_code == 200:
                # Create a ZipFile object from the downloaded content
                with zipfile.ZipFile(io.BytesIO(response.content), 'r') as zip_ref:
                    # Specify the directory where you want to extract the contents
                    extraction_path = "./"

                    # Get the list of files in the zip
                    file_list = zip_ref.namelist()

                    # Check if any file in the zip already exists in the extraction directory
                    existing_files = [file for file in file_list if os.path.exists(os.path.join(extraction_path, file))]

                    # Extract all the contents of the zip file to the specified directory
                    if not all(os.path.exists(os.path.join(extraction_path, file)) for file in file_list):
                        zip_ref.extractall(extraction_path)

            else:
                print("Failed to download the file.")

    code_eval = load("code_eval")
    dataset = load_dataset("openai_humaneval")
    print(f"Dataset size: {len(dataset['test'])}")
    for model_folder in tqdm(os.listdir(results_folder)):
        test_cases, predictions = [], []
        for sample in dataset["test"]:
            with open(f"{results_folder}{model_folder}/{sample['task_id'].replace('/', '_')}/0.py") as f:
                prediction = f.read()
            test_cases.append(f"{sample['test']}\ncheck({sample['entry_point']})\n")
            predictions.append([prediction])

        pass_at_k, results = code_eval.compute(references=test_cases, predictions=predictions, k=[1], timeout=60*5,
                                               num_workers=multiprocessing.cpu_count())
        print(model_folder, round(100 * pass_at_k["pass@1"], 1))
        for result in results.values():
            assert len(result) == 1
            _, result = result[0]
            assert result["result"].startswith(("passed", "failed")), result["result"]
  ```
</details>

Hi @JoaoLages it is not expected to work if you want to use the additional tests with other toolings :(.

Specifically, the output samples from the release are the LLM provided solutions -- they would not embedd the test-cases since that would be too large to repeatively include such a large amount of tests. Meanwhile, our testing logic is NOT like the original HumanEval where the pass or fail was determined by the process status...

Instead, to use `evalplus`, you are expected to use `evalplus.evaluate` which will perform just-in-time differential testing (aka cross-checking) over our test inputs -- yeah we don't store the test-code as HumanEval etc. but use the test inputs to be more structural. Nonetheless, one could try to convert the format/schema of our [dataset](https://github.com/ganler/release/releases/tag/humanevalplus) to that of HumanEval which might be compatible with other evaluation tools.

Got it, thanks for your help! 🙏 
As a final note, can you tell me where you got the LLM samples from [this release](https://github.com/evalplus/evalplus/releases/tag/v0.1.0)? 

@JoaoLages These samples are generated from scratch using code from https://github.com/evalplus/evalplus/tree/master/codegen

A sample command can be:

```bash
python codegen/generate.py --model ${MODEL} --bs 1 --temperature 0 --n_samples 1 --resume --greedy --root ${SAMPLE_ROOT}
```

Happy Holiday! :D

Thank you and happy holidays! 🎅 🎄