Repository: evalplus/evalplus
Issue: [TODO] Evaluating Code Llama
Root cause: Unimplemented Feature Gap

Code Llama is availabel on https://github.com/facebookresearch/codellama and I have gotten the access.

I am trying to install and evaluate the 7b version first.

Inference implemented in 05990898e19bd2477187738b8e08d0719a9a6265. Waiting for resources to run it.

CodeLlama 7B and 13B results are updated on https://evalplus.github.io/

CodeLlama 34B somehow did not generate one input after 10 minutes. Am debugging it right now.

> Inference implemented in [0599089](https://github.com/evalplus/evalplus/commit/05990898e19bd2477187738b8e08d0719a9a6265). Waiting for resources to run it.

Fixed a few bugs in fc10dc72addb508dcd0084133ad425972887d709 and now it should at least work for 7b and 13b.

> I can try to run your codellama evaluation code and confirm results

Hi Yacer thanks for the support. 

For you and others who are interested in reproducing the results, feel free to try the scripts below after a git clone install:

```bash
export PYTHONPATH=$(pwd)

# 7b
CODE_LLAMA_ROOT=/path/to/7b-python-model torchrun --nproc_per_node 1 codegen/generate.py --model code-llama-7b --bs 1 --temperature 0 --n_samples 1 --resume --greedy --root /path/to/root
python tools/sanitize.py --folder /path/to/root/code-llama-7b_temp_0.0
python evalplus/evaluate.py --dataset humaneval --samples /path/to/root/code-llama-7b_temp_0.0-sanitized --parallel $(nproc)

# 13b
CODE_LLAMA_ROOT=/path/to/13b-python-model torchrun --nproc_per_node 2 codegen/generate.py --model code-llama-13b --bs 1 --temperature 0 --n_samples 1 --resume --greedy --root /path/to/root
python tools/sanitize.py --folder /path/to/root/code-llama-13b_temp_0.0
python evalplus/evaluate.py --dataset humaneval --samples /path/to/root/code-llama-13b_temp_0.0-sanitized --parallel $(nproc)

# 34b
CODE_LLAMA_ROOT=/path/to/34b-python-model torchrun --nproc_per_node 4 codegen/generate.py --model code-llama-34b --bs 1 --temperature 0 --n_samples 1 --resume --greedy --root /path/to/root
python tools/sanitize.py --folder /path/to/root/code-llama-34b_temp_0.0
python evalplus/evaluate.py --dataset humaneval --samples /path/to/root/code-llama-34b_temp_0.0-sanitized --parallel $(nproc)
```

Of course you will need 4 GPUs w/ 16GB+ memory and have the corresponding models installed. Also see https://github.com/evalplus/evalplus/blob/master/codegen/model.py#L637

The stuck seems to be related to a PyTorch thread here: https://github.com/pytorch/pytorch/issues/22259#issuecomment-1439447697 since the machine is using the same NCCL version.  

Bug solved by setting `NCCL_P2P_LEVEL=NVL`. (source: https://github.com/pytorch/pytorch/issues/73206#issuecomment-1054442959)

All updated here! https://evalplus.github.io/leaderboard.html

Sorry. There are some dependencies required in https://github.com/evalplus/evalplus/blob/master/requirements-llm.txt


> I think I got confused, reading model.py, it seems to be tailored to benchmark models directly downloaded from codellama. What if we want to evaluate the codellama from hf ?

Yes it is. To be consistent with the original CodeLlama usage as much as possible, I directly used followed the code in their official repo.