Repository: evalplus/evalplus
Issue: [Bug] 0.2.1 may fail with some cases on Mbpp evaluation while 0.2.0 works fine
Root cause: Validation Gap

I run evalplus on HumanEval and Mbpp using examples from Magicoder's git repo: https://github.com/ise-uiuc/magicoder/tree/main/experiments. Days before it worked OK, but this error occurred with evalplus==0.2.1 (upgraded).

Since the script used a local evalplus folder, this had been added to the shell env:
```
export PYTHONPATH=/home/username/evalplus
```

Here is the original error info:
```
...(logs)
Sanitized: Mbpp/113 (line 65 in /home/username/codeEval/assessment/evalplus-trained_model-mbpp.jsonl)
65it [00:00, 598.90it/s]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/username/evalplus/tools/sanitize.py", line 6, in <module>
    main()
  File "/home/username/evalplus/evalplus/sanitize.py", line 247, in main
    Fire(script)
  File "/home/username/miniforge3/envs/lf/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/miniforge3/envs/lf/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/username/miniforge3/envs/lf/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/evalplus/evalplus/sanitize.py", line 220, in script
    new_code = sanitize(code=old_code, entrypoint=function_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/evalplus/evalplus/sanitize.py", line 166, in sanitize
    reacheable = get_function_dependency(entrypoint, call_graph)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/evalplus/evalplus/sanitize.py", line 84, in get_function_dependency
    for neighbour in call_graph[current]:
                     ~~~~~~~~~~^^^^^^^^^
KeyError: 'empty_dit'
```

I checked the dataset file and they were the same as before. I renamed the function in Mbpp/115 as empty_dict and another similar error just showed up, so I guessed the problem was not caused by their names.

And I tried to switch the local evalplus to the elder version 0.2.0 (by resetting to commit ae74712) and things worked fine just as before.
HumanEval has not met such an issue for now.

Thanks @yucc-leon 

I believe this error was brought by the latest version (v0.2.2) on GitHub.

Temporarily, you may want to make sure of using `v0.2.1` by uninstalling evalplus first:

```
pip uninstall evalplus
pip install evalplus==0.2.1
```  

Can you also provide the code that runs into this issue? Our HEAD branch is now using a treesitter-based sanitizer (https://github.com/evalplus/evalplus/pull/132) which may have bugs because it's new. also cc @JialeTomTian 

> Thanks @yucc-leon
> 
> I believe this error was brought by the latest version (v0.2.2) on GitHub.
> 
> Temporarily, you may want to make sure of using `v0.2.1` by uninstalling evalplus first:
> 
> ```
> pip uninstall evalplus
> pip install evalplus==0.2.1
> ```
> 
> Can you also provide the code that runs into this issue? Our HEAD branch is now using a treesitter-based sanitizer (#132) which may have bugs because it's new. also cc @JialeTomTian

Thanks for replying. Here is the code (modified from Magicoder's repo:
```
export PYTHONPATH=/home/workspace/evalplus # using 0.2.0 commit

MODEL_KEY=deepseek-ai/deepseek-coder-6.7b-base
MODEL=/home/workspace/modelname
DATASET=mbpp
SAVE_PATH=assessment/evalplus-$(basename $MODEL)-$DATASET.jsonl
SANITIZED_PATH=assessment/evalplus-$(basename $MODEL)-$DATASET-sanitized.jsonl

CUDA_VISIBLE_DEVICES=0 python -m text2code \
   --model_key $MODEL_KEY \
   --model_name_or_path $MODEL \
   --save_path $SAVE_PATH \
   --dataset $DATASET \
   --temperature 0.0 \
   --top_p 1.0 \
   --max_new_tokens 512 \
   --n_problems_per_batch 24 \
   --n_samples_per_problem 1 \
   --n_batches 1

python -m tools.sanitize --dataset $DATASET --samples $SAVE_PATH
evalplus.evaluate --dataset $DATASET --samples $SANITIZED_PATH
# evaluate also using pip install evalplus==0.2.0
```
Here providing the model's output file:
[evalplus-modelname-mbpp.jsonl.zip](https://github.com/evalplus/evalplus/files/14901056/evalplus-modelname-mbpp.jsonl.zip)



This should have been fixed by 8e2d7d469a012010cd9900d4be0e6a77567344c9

You can either use an old version or try:

```bash
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

Feel free to reopen if errors are found.