Repository: evalplus/evalplus
Issue: Runtime error
Root cause: Validation Gap

Hi,

I had been using Evalplus for a while, it was working fine, last used Oct 9th. But today, I went to re-run some old tests, but it gave me an error.

This is the command I am using !`HUMANEVAL_OVERRIDE_PATH=$(realpath quick-start.jsonl) $YOUR_EVALPLUS_COMMAND evalplus.evaluate --dataset humaneval --samples samples.jsonl`

This is a snapshot of the samples.JSON file
![image](https://github.com/evalplus/evalplus/assets/83566027/e598b19f-479e-4b1c-a8ee-de321db1f004)

But I am getting this error.
```
Traceback (most recent call last):
  File "/usr/local/bin/evalplus.evaluate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 273, in main
    evaluate_humaneval(args)
  File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 132, in evaluate_humaneval
    problems = get_human_eval_plus(mini=flags.mini)
  File "/usr/local/lib/python3.10/dist-packages/evalplus/data/__init__.py", line 161, in get_human_eval_plus
    plus = {task["task_id"]: task for task in stream_jsonl(plus_path)}
  File "/usr/local/lib/python3.10/dist-packages/evalplus/data/__init__.py", line 161, in <dictcomp>
    plus = {task["task_id"]: task for task in stream_jsonl(plus_path)}
  File "/usr/local/lib/python3.10/dist-packages/evalplus/data/__init__.py", line 68, in stream_jsonl
    yield json.loads(line)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 10 (char 9)
```
 

Hi, can you share the `samples.jsonl` for me to debug? Thanks!

Hi thanks for getting back. I was able to resolve the issue, it was a syntax Error. 😄 

Unfortunatioy I am getting a new error 😢 
`Computing expected output...
Expected outputs computed in 0.00s
Traceback (most recent call last):
  File "/usr/local/bin/evalplus.evaluate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 293, in main
    evaluate(args)
  File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 144, in evaluate
    expected_output = get_groundtruth(problems, dataset_hash, [])
  File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 69, in get_groundtruth
    with open(cache_file, "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/evalplus/6564efad89a88db2059a3b9f9ef0933f.pkl'
`

It seems the cache directory was not there. But they should have been created when we first download the dataset in https://github.com/evalplus/evalplus/blob/master/evalplus/data/humaneval.py#L23

Can you start a fresh container and retry? You can also manually create a cache directory (`mkdir /root/.cache/evalplus/`) in the container. I will also try to make the code more robust by always performing CACHE_DIR checking.
 

Thank you for your help. 

Fixed in `ae7f2127f9052740cb15d073b9e8d343310222a8`. The docker image is released for each version and the changes made so far since the last version `v0.2.0` is not yet enough, so I am not gong to update the image release right now. But you can also just enter the image and perform the following operations:

```bash
cd /evalplus
git pull orgin master
pip install .
```

or simply:

```bash
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

Thanks for posting the issue!